Polkit is how a lot of things on modern Linux systems decide whether or not to let people do privileged operations. A while back I investigated making Polkit's administrative access operate like su does, where you authenticated using the root password and only some people could do it. Today I said something related to that on the Fediverse:

It certainly is a choice for Linux's Polkit to decide that there is no way to say "this login does not have administrative access period, no matter what".

(Your only choice is to force people to try to authenticate using the 'password' of a locked account, which will always fail.)

This turns out to not quite be true (you can force people to fail authentication without a password prompt), but getting there is a journey.

Operations using Polkit can require "authentication by an administrative user", and Polkit lets you control what an 'administrative user' is. This is done by writing a JavaScript function that 'specifies what identities may be used for administrator authentication' (possibly in general or possibly for a specific operation). You pass this function to Polkit's polkit.addAdminRule() and then it later gets called and returns something. There can be multiple functions; they're called in order of registration (which is generally lexical order of the .rules files) and the first one to answer wins.

As we can see in _runAdminRules, what an answer is is a return value that JavaScript 'if (...)' will consider to be true (a truthy value). This value is expected to be an array of strings, and .join(",") will be called on it. If the result of that can't be turned into a string, there will likely be some sort of exception raised (also); otherwise, the string value is split again and Polkit's C code tried to turn each piece into an identity. If no pieces are valid identities like 'unix-user:0' or 'unix-group:wheel', the code explicitly falls back to authenticating against the root user. Then, at a higher level, groups and netgroups are expanded to users, and if they expand to nothing, there's once again an explicit fallback to root. Polkit is really insistent that everyone be able to authenticate as some administrative user and it doesn't have any particular notion that some people can't do that at all.

The obvious way around all of this mess is what I did in the original entry, where I returned an existing login with a locked password. This caused Polkit to ask you to authenticate as that login, which would always fail. For operations people weren't supposed to do anyway, this is acceptable even if it's unaesthetic. Now that I've read the code I've come up with a nominally better but more alarming way, which is to have your admin rule function return '["unix-user:NNN"]' for some UID that definitely doesn't exist and never will. This works today because Polkit doesn't validate that the UID exists before it attempts to use it for password authentication; when Polkit finds that the UID doesn't exist, it's too late for it to do anything but fail. People trying this will get a pleasant message like:

; run0 echo hi
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ====
Authentication is required to start transient unit 'run-[...].service'.
Failed to start transient service unit: Access denied

An 'Access denied' error is about as clear and as good as you can do.

(If you use '["unix-user:nosuchperson"]', Polkit does validate the login name and rejects the identity, leaving you with the root fallback.)

However, I suspect that this is a somewhat dangerous approach. Someday Polkit may get smarter about detecting nonexistent Unix UIDs in 'unix-user:...' things in admin rules and start rejecting them early, causing it to fall back to demanding the root password. An existing user with a locked password that people can never enter is less aesthetic but safer in the long term.

(With that said I doubt Polkit is going to change this behavior in the kind of bugfix release that might be picked up by a specific release of a Linux distribution, so given that it works in Ubuntu 26.04 today it's probably going to keep working for the lifetime of 26.04. And you can use very large UID numbers to make sure you'll never have an actual account of that UID.)

Sidebar: What doesn't work and why

Since I went through this exercise, both testing things and reading the combined source code:

• Returning '[]' from your admin function returns an empty string to the Polkit C layer, which then fails to be converted to an identity.
• Both '["unix-user:nosuch"]' and '["unix-group:nosuch"]' fail at the Polkit conversion from string(s) to identities.
• '["unix-group:emptygrp"]' and '["unix-group:NNNN"]' for a GID that doesn't exist both fail at the Polkit expansion of groups to Unix logins; with no logins generated, Polkit follows the second fallback to root.
• Returning 'polkit.Result.NO' from your admin function actually returns the string "no", which also fails to turn into an identity (see init.js).

A nonexistent Unix UID as '["unix-user:NNNN"]' is the only option that threads all of these needles; it's accepted in the string to identity conversion step, and it's not currently rejected at the point when Unix groups are expanded (which would be the natural point to put such a check, if Polkit wanted to).

Today I got a useful reminder about how you should always restart programs and daemons after changes, or at the very least explicitly check to make sure that they've picked up changes you recently made. This lesson came in the form of a couple of hours of downtime for Wandering Thoughts, where it (mostly) responded with HTTP 500s between 08:38 and 10:53 today. I didn't make the change that triggered this at 08:38 or any time immediately before then; I made the change late last night.

For reasons beyond the scope of this entry, DWiki (the software behind Wandering Thoughts) runs either as an Apache CGI or as a persistent daemon process (which a front-end CGI communicates with). Normally it runs as a CGI for convenience, but it switches to the daemon mode when the host server is loaded enough; one time this usually happens is during overnight host backups. Once the load drops low enough and enough time has gone by, the daemon quietly shuts down and DWiki reverts to running as a CGI (typically it runs this way most of the time).

When I made the change late last night, I 'tested' it by seeing if Wandering Thoughts still worked. It did, but only because DWiki was running as a daemon at the time, which freezes everything on startup and so hadn't picked up my change. I forgot about this possibility because I usually make changes at times when DWiki is running as a CGI so my changes are picked up immediately. Instead, things only got quiet enough on the host at 08:38, triggering daemon shutdown, causing the next run to finally pick up the change and surface the problem.

Of course always remembering to restart (or 'reload') daemons is easier said than done, especially in situations where it's only necessary some of the time, not all of the time. Programs automatically picking up some changes is very convenient until you lose track of which changes are automatically picked up and which aren't (and when), and sometimes this can be obscure. This learning experience about DWiki changes will probably stay with me for a while, but I want to think about ways to make this more reliable without adding too much code complexity.

(DWiki already has a flag file option that can be touched to signal the daemon to shut down, which I currently point at a file used only for that. An obvious improvement would be to point the option at the configuration file, so any configuration file edits automatically cause the daemon to shut down and maybe restart.)

I recently wrote about our authentication versus authorization problem, where we've been forced to shoehorn a bunch of things that are actually about authorization into our authentication systems. An additional realization is that some of what we might want isn't even about what an account is authorized for (its authorizations), it's about an account's capabilities.

Let's make this concrete. One of the things we considered recently was whether we could easily create special limited 'guest' accounts that couldn't use our email system at all. This went beyond not being able to use our IMAP system or send email via authenticated SMTP (which is authentication or authorization) to not being able to receive outside email to '<user>@<us>'. This isn't something that can naturally be covered by 'authorization', because the account involved isn't trying to do anything when our email system receives a message for it; there's no authentication for the account and thus no authorization to be derived from it.

(To me this is the dividing line between authorization and capabilities. Authorization is checked at the point where an account is actively trying to do something. Capabilities are things that you can check even outside of an attempt by the account to do something.)

Thinking of things as capabilities (in the abstract) means that you need some way to query or check an account's capabilities outside of an access attempt, and probably some way to get a list of all of the accounts with a given capability (such as 'receive outside email'). But it also means you don't have to fit every last piece of information about what an account is allowed to do into an authorization system, or especially into what your particular authorization system allows you to express.

(You could record whether or not Unix logins can receive email by putting them in 'yesmail' or 'nomail' Unix groups, depending which is more common, but this is probably not a particularly good use of Unix's basic authorization system. Especially if there's no actual Unix filesystem permissions involved and the groups are being used only as markers, so you've turned your /etc/groups into a general attribute tag system for logins.)

If you can query capabilities at any time they can probably be used for authorization decisions as well, so in a sense capabilities are a superset of authorizations. The mechanics may be different, though, in that some things will expect to receive (or generate) authorization information at the same time as they perform authentication, not to make a separate query to determine authorization status once someone has been authenticated.

For me, this is an especially useful insight if most of your authorization is actually being decided based on whether or not people can authenticate to some particular service (as is our case). Now I have a more general model for thinking about what an account can do, what we want to record about that, and how we want to express some things. Also, now I have a dividing line I can draw between something we can maybe try to express through authentication and authorization and something we need to step outside of that model to handle.

(One comment.)

Over on the Fediverse, I took part in a small conversation about the lacking state of tools for making (structured) changes to code, what you'd often call refactoring. Editors have long had some degree of support for this, but general support is spotty, especially in command line tools. There are things that exist, but they tend to be experimental; for example, the Go developers sometimes use rf to make structured, deterministic changes to code (such as this split of SSA rewrites into packages, which used rf to make a series of structured changes).

When I was mentioning rf, it occurred to me that we increasingly have tooling that could probably be used to create tools for this. Tree-sitter provides high quality concrete syntax trees for many languages, often with meaning annotated, and LSP servers provide specific code intelligence about the meaning of identifiers and so on, also for many languages. Given a tree-sitter syntax tree and a LSP server, in theory you have enough information to do various sorts of code reconstruction in a relatively language independent way. Some amount of this can also be driven directly through the LSP server, since some of them directly support various sorts of refactorings and changes.

(It will help a lot if you also have access to a language formatter, for example through the LSP server, so you can have the formatter fix up your edited text to the language's standards afterward.)

Effectively this would be combining what a modern editor already does with tree-sitter grammars and a LSP client for interactive (structured) edits, but driven through either a command line or through some structured description of the changes you want (in the style of rf but perhaps in a file). There's at least one command line LSP client (lsp-cli, also), but it seems to be focused on inspection rather than changes (which is reasonable, most changes are a lot more complex than merely querying a language server).

I think the usual view is that you want to drive these changes through an editor so you can see the result, but in today's environment of pervasive version control the tool could perform the change and then you'd get a VCS diff to see the result. There are already language specific tools that can do some of fixes for you; I know of ones for both Go and Python, and sometimes checkers and other diagnostic sources can export fix information that a program (most often an editor) can use to automatically apply them.

(Go has a rich ecology of them because early on it built a framework for doing this, which most famously surfaced in gofmt but which also powers a bunch of analysis and sometimes rewrite frameworks, such as "go fix".)

One of the things that's said regularly is that authentication is not authorization, and there's a difference between the two. This can all sound abstracted until one day you realize that you've wound up in an awkward place because of it and because your environment smashes them together. I say this because I recently realized that one of our practical problems is the result of exactly this lack of difference in our environment.

We operate what's ultimately a quite traditional Unix based multi-user environment (cf), along with an assortment of related services like an IMAP email server and the ability to send and receive email to '<login>@<us>'. Ultimately all authentication that people do comes down to a global /etc/passwd, although it gets transformed in a bunch of ways for things like our web based OIDC authentication. Since we started with Unix machines, we didn't really have a separate concept of authorization; if you could log in, you were authorized (more or less).

All of this was fine in an environment where we had only one sort of account (and login), an account that could do everything. Such accounts appeared in the global /etc/passwd, appeared in the local /etc/passwd on every machine with NFS mounts, could be used for IMAP authentication (through the IMAP server's /etc/passwd) and web SSO authentication and so on. Then as time went by we started wanting special sorts of accounts that were not supposed to have this sort of full access. Accounts that were only there to use our VPN servers, accounts that were only there to authenticate to an outside service (which would send them email from time to time so they had to receive email but we didn't want them to send it from their account), accounts that hadn't been fully activated yet and so shouldn't be able to register machines for network access (or do some other things), accounts that should only be able to use a few systems, and so on.

Because nothing we'd built had a separate idea of 'authorization', most of this had to be wedged into an increasingly awkward set of hacks around who was in what version of our global password database (and in what state). In a few cases we could use Unix groups (or Apache groups), and for a bunch of machines we had to rely on special shells (despite the flaws). Because of the limitations of special shells, some of the limits on restricted accounts are partially illusory and could be worked around by a determined person (or by accident). It's been particularly difficult to put fine grained restrictions on what people could do through our web-based services because almost all of the authorization is done externally, for example by Apache applying a restriction to a directory of simple CGIs.

This ad-hoc approach has a variety of problems. Any time we want to apply a new sort of restriction or restrict a new sort of system, we have to figure out how to thread it through our environment, or what we can actually restrict while still allowing what's needed. Since there's no central database of what sort of account (or who) is allowed to do what, we can easily lose track of what a particular type of account is allowed to do or is blocked from, and in fact working it out can involve going to specific machines and specific software systems to refresh our memory of how they work (including, for example, how the Apache specific password and group files are built when we're using HTTP Basic Authentication or even OIDC with Apache group files).

(In some cases we've resorted to creating synthetic groups in the Apache group file that are based on, for example, the Unix shell that an account has.)

This has also constrained what sort of restricted accounts we can provide easily (or at all). In the recent case that crystallized my realization, we were asked if we could do relatively limited 'guest' accounts with some new restrictions and the answer was not really. Technically we could have implemented some of it, but it would have meant adding even more complexity to some already complex systems. The whole discussion more or less rubbed my nose into how it would be quite useful for us to have a clear distinction between authentication and authorization.

(I don't know what that distinction would look like in practice. It's easy to think about the abstraction of 'is authenticated login X allowed to do Y', but it's a lot harder to build a manageable system that can answer that question in the various contexts we need to do it.)

(3 comments.)

Today I got around to upgrading my office desktop from Fedora 43 to Fedora 44. As usual I did this through the 'dnf distro-sync' live upgrade process. Normally this goes okay (and it did work fine in my test virtual machine), but this time around there was an unnerving problem:

Just what I want to see during a dnf distro-sync upgrade from Fedora 43 to 44, specifically during the posttrans actions:

Transaction failed: Rpm transaction failed.

Why? Who knows, dnf/RPM doesn't say. How do you recover? Who knows. What does 'failed' mean here when everything was installed and seems to be working? etc.

Time to pick through 'rpm -V' nervously, I guess.

A 'dnf distro-sync' upgrade first upgrades, installs, and sometimes downgrades all new packages from the new Fedora release, then it removes all of the now-replaced old packages, and then at the end it runs a series of 'postrans' end-of-transaction scripts which do various things like build the kernel initramfs images in /boot. Then normally DNF tells you that everything succeeded. This time it instead printed some output and then exited with that message. Specifically, dnf's output ended with:

[...]
>>> Running %triggerpostun scriptlet: systemd-0:259.8-1.fc44.x86_64
>>> Finished %triggerpostun scriptlet: systemd-0:259.8-1.fc44.x86_64
>>> Scriptlet output:
>>> Job failed. See "journalctl -xe" for details.
>>>
Transaction failed: Rpm transaction failed.

Naturally I jumped to the assumption that the systemd triggerpostun script's output meant that it had failed and this had caused the entire transaction to 'fail'. I spent some time trying to follow this through systemd logs and so on, but in the end I think this was a red herring.

I put 'fail' in quotes because inspection of the system afterward showed that all of my packages had been upgraded to their Fedora 44 versions and the old Fedora 43 ones were gone. However, this left me uncertain about how many of the various end of transaction scripts had actually run and whether my system now had incomplete or not-done work that was necessary for it to work properly.

I'm not sure that the dnf5 logs contain enough information to tell for sure (at least for people who aren't familiar with dnf5 internals), but the default Fedora dnf log non-retention policy didn't help. Partly this was because I only started desperately looking through the logs after I'd tried to 'redo' or 'replay' the transaction (with 'dnf history redo' and 'dnf replay' respectively), neither of which worked and neither of which I understand. I was rather lucky that partial logs from the distro-sync upgrade were still present, and those logs let me determine that a lot of scriptlets had run after the systemd one, so probably everything had run. The logs also revealed that despite the scary message, the systemd scriptlet had exited successfully.

INFO RPM callback start %triggerpostun scriptlet "systemd-0:259.8-1.fc44.x86_64"
INFO [scriptlet] Job failed. See "journalctl -xe" for details.
INFO RPM callback stop %triggerpostun scriptlet "systemd-0:259.8-1.fc44.x86_64" return code 0

For various reasons, I run all of my live Fedora system upgrades under the venerable script(1), so I can capture full output, and also I copy any alarming snippets I see into my own records. This turned out to be extremely useful because I was able to go back to both sources and find a very confusing message:

[...]
>>> Running %post scriptlet: pax-0:3.4-49.fc44.x86_64
>>> Non-critical error in %post scriptlet: pax-0:3.4-49.fc44.x86_64
>>> Scriptlet output:
>>> alternatives version 1.33 - Copyright (C) 2001 Red Hat, Inc.
>>> This may be freely redistributed under the terms of the GNU Public License.
>>>
>>> usage: alternatives --install <link> <name> <path> <priority>
[... usage message omitted ...]
>>> [RPM] %post(pax-3.4-49.fc44.x86_64) scriptlet failed, exit status 2
[...]

(All of this was buried in the output of upgrading, installing, downgrading, and removing 14340 packages, some of which produced random messages during this process.)

Despite the 'non-critical error' bit, this was the only thing that looked odd, and also my test virtual machine didn't have the 'pax' package installed. So my tentative conclusion is that this pax scriptlet failure blew up the entire transaction, but also the 'failure' of the transaction was ultimately harmless and my system is properly upgraded to Fedora 44.

Well, mostly, because now I've discovered that 'dnf list --installed' claims that all of my Fedora 44 are from a repository that is listed as '<unknown>', instead of from 'fedora' or 'updates' as you'd expect (and which is what I see on my test virtual machine). I'm not sure how one would go about fixing this, especially since 'dnf distro-sync' doesn't see anything to do. Presumably it will fix itself over time as more updates come out and then I eventually upgrade to Fedora 45 (assuming that doesn't have a transaction failure too).

(I'm not sure I want to try to 'dnf reinstall' everything, or really anything. In theory this is a problem that can be fixed entirely through some sort of database manipulation; DNF can easily determine that my packages come exactly from various configured repositories.)

I haven't been enchanted with dnf5 so far (1, 2, 3), and this experience hasn't improved my feelings. In the one case where you really, really want a complete, verbose, and comprehensible error message that tells you exactly what went wrong, why the transaction failed, what was undone, and what you can do next, dnf5 (and RPM) completely fell short.

(It's possible that dnf5 is doing the best it can with the information that the RPM library and Python bindings give it, but that still doesn't absolve Fedora. DNF and RPM are effectively conjoined at this point, so it's a collective responsibility, and DNF itself could have at least said something more.)

(6 comments.)

A lot of things these days can report useful information about their status and health. Or at least their theoretical status and health. As we've seen, things like server BMCs may be reporting outdated hardware inventories. Today I got another vivid illustration of that, when Toronto had what you could call a significant weather event with some power flickers in my area, and I made an unhappy discovery about my home UPS:

TIL that my home UPS has recently become a home power bar or maybe a home line conditioner, despite claiming that its battery is fine. Well, it’s six years old, but on the other hand it worked in June.

(Specifically it worked on June 1st early in the morning, for a few seconds, and before that on May 23rd.)

This particular UPS is a bit over six years old, and has been periodically useful during fortunately infrequent brief power outages. But this time, when the power flickered briefly everything connected to the UPS powered off, which is the reverse of what I'd like. After a couple more power flickers, the UPS stayed off for long enough that I disconnected everything and switched back to my pre-UPS power bar.

I have monitoring of the UPS set up on my home desktop, and all during the power events both it and the UPS's front panel display claimed that the UPS had 100% battery, the battery was at 24 volts, and so on (even after a power flicker had caused the UPS and everything attached to it to shut off, then power came back and everything turned on). But once I disconnected everything from the UPS and was able to get the UPS to do a proper self test, it abruptly switched to reporting 6% battery charge, which is a much more realistic number for something that fell over more or less the moment it needed to supply battery power (and there are some indications that even that's misleading; for example the nominal charge level will drop from 20% to 5% in a few seconds even with nothing connected to the UPS).

I knew that battery charge reporting was somewhat unreliable for Li-ion batteries, but I don't think I've seen this before in regular batteries and our rack UPSes at work have always been reasonably good about reporting that their batteries needed replacement. Unreliable reporting of battery status matters for more than surprising me when the power went out; initially it made me think that the entire UPS unit needed to be replaced, rather than just the batteries (probably) being worn out.

(I was willing to believe that the batteries were dead, but a UPS that firmly reported dead batteries as being completely good seemed like a UPS that might have additional problems that had degraded its ability to work right in general.)

PS: In the long run, I should plan to periodically test that the UPS can support the load when line power goes off, either through its own self test or by manually switching it away from line power with a front panel switch. Obviously I'd better do this at a time when I can afford to have my desktop abruptly lose power.

Sidebar: Did the UPS make things worse?

None of the power flickers lasted long enough to cause the usual run of kitchen appliances to reset their clocks, and there were a few more power flickers after I'd moved everything to the power bar and nothing rebooted. Modern desktop PSUs apparently normally have a hold-up time of 16 milliseconds or more (it's apparently part of the ATX specification, although I haven't looked to be sure), which is the length of time the PSU should keep supplying power after the line power goes away. So these power flickers seem to have been very brief, but apparently they were still long enough that the UPS tried to switch to battery power (and then had battery power fall over).

If the UPS had not detected power loss and tried to switch, it seems possible that everything would have survived fine. On the other hand, I wouldn't know that the UPS had problems.

(The UPS definitely made things better back in June and May.)

(6 comments.)

We have two sorts of multi-user Linux machines; there are ones for general purpose logins and normal Unix stuff, and ones that are specifically for heavy duty compute. Many years ago, we began using the Linux kernel's strict memory overcommit mode on both sorts of machines. We soon determined this wasn't a good idea on the login servers, but left it on the compute servers for many years afterward even when it caused occasional problems (also, also).

(On the login servers we wound up setting systemd memory limits on each person's total memory use, but this isn't something we could do on the compute servers, where one person should be allowed to use all of the RAM if it's not otherwise in use.)

These days, another plausible way to limit the total amount of memory people can use on your servers is to set systemd memory limits on user.slice. This limits actual RAM usage (from all sources), not the amount of memory people may theoretically ask the kernel to provide, but usually this is the more relevant limit. When I started considering this, I looked into how much RAM people were actually using on our compute servers as compared to the committed address space and discovered things had changed since the past.

Once upon a time, a long time ago, it was generally the case that compute programs on our compute servers would actually use most of the (large) amounts of memory they asked for. This made strict overcommit reasonable on these machines, because if a big compute job asked for 32 GB of RAM, it was probably going to use it. That's no longer the case for the kinds of things people run on our servers; the committed address space is far larger than the (current) RAM usage for all of user.slice (I ended up collecting some metrics). We also have evidence some programs ask for much more RAM than they can use, such as all of the RAM in the server (in one extreme case). I don't know why this shift has happened (all I can do is speculate about, for example, shifts in common programming languages to use in compute intensive things), but it clearly has. My data showed several times more committed address space claimed than RAM in use, and it was clear that people were being artificially limited by our choice of strict overcommit mode.

Given this, we decided to switch off strict memory overcommit on our compute servers and instead put a MemoryMax= setting on user.slice to leave a few gigabytes for system services (cf). This hasn't solved all of our problems but I think it makes things better, and people are definitely using more RAM than before (one of our compute servers is currently well over the point where it would have hit strict overcommit limits).

In general I think these cgroup-based limits make more sense in a modern Linux kernel environment than strict memory overcommit does, unless you have an unusual situation. They also make the memory limits on our general purpose compute servers operate the same way that memory limits do in our SLURM environment, where the SLURM daemons use cgroup memory limits on individual jobs (which are each put in their own cgroup to enable this).

PS: Overcommitting virtual memory is sensible in general. We pervasively over-allocate through the programming stack in order to trade memory for speed.

Over on the Fediverse I had a question:

GNU Emacs people who write Python using venvs, what's the state of the art for getting linters, LSP servers, and so on started with the right paths/etc set up for a particular project's venv? Is it (still) direnv + the envrc package, or something else (.dir-locals.el with a exec-path setting, maybe)?

For my sins, straightforward venv activation stuff won't work in my GNU Emacs + Python setup because often I'm developing code outside of the venv with the supporting packages for that code. For reasons, yes I know this is odd, it's our (my) way.

After playing around with this for a bit I think I have a recipe that works, although I'm not sure it's completely correct.

To start, you want to do the brute force thing and install copies of every LSP server and tool you want to use into your project's virtual environment. This isn't absolutely required for some tools, but we're already throwing disk space at the problem (that's partly what venvs are about) and it's the easiest way to make Python-based tools like the Python LSP server happy.

Then we have two choices. First, you can painstakingly activate the virtual environment every time before you run GNU Emacs to edit files in the project. This will set $PATH and $VIRTUAL_ENV, picking up and generally configuring your installed tools properly, but speaking from personal experience this is kind of a pain (plus it means you may need to run multiple instances of GNU Emacs), and sooner or later I forget. Second, you can use direnv and the GNU Emacs envrc package to automatically configure things in the relevant buffers.

To use the direnv based approach, each project's root needs a .envrc file that exports a $VIRTUAL_ENV with the appropriate top level venv directory and a $PATH that puts the venv's bin/ directory on it (probably first). Then you enable that .envrc in direnv, install envrc into GNU Emacs from MELPA, and add something like the following to your .emacs:

(use-package envrc
  :defer t
  :if (executable-find "direnv")
  :hook
  (python-mode . envrc-mode)
  (python-ts-mode . envrc-mode)
  )

You can set envrc's global mode if you want, but I opted to be narrow. If I want environment variables set outside of editing Python files, I'll enable it in more modes.

So far, all of this has worked for me in GNU Emacs. The projects (ie, directory trees) where I have a .envrc set up run the right versions of LSP server(s) and tools, and I can edit files from multiple projects in the same Emacs session and have everything work out because envrc makes these settings buffer-local, so each project can get the right LSP server and other settings.

Setting $PATH and $VIRTUAL_ENV has a varying set of effects. For Python based tools installed in the venv, it means that they'll run from the venv using the venv's Python and the venv's Python packages available to be introspected and so on; this covers pylsp and mypy (and I think basedpyright if you want to use a very strict LSP server). The effects on LSP servers not written in Python and other tools varies. Pyrefly currently determines the Python import path by running the 'python3' that's on your $PATH and dumping its 'sys.path', while ty looks at $VIRTUAL_ENV (cf).

(Currently, I believe that ruff can be installed globally because as far as I can tell, it doesn't look at Python imports at all. Installing it into your venv is relatively cheap insurance, though.)

In theory you can get some of this by merely setting $PYTHONPATH to <venv>/lib/python3.NN/site-packages, but I wouldn't trust that to be fully functional and without side effects. It's better to set the environment variables in your .envrc, which more or less duplicates what the venv 'activate' script does. I have set $PYTHONPATH in addition to the other variables as the easy way to make a small pipx-based venv work (in a simple case) without having to inject a bunch of additional tools into it.

(All of this is unlikely to work remotely over Tramp.)

PS: You may or may not want to hook direnv up to your shell. I haven't, partly because it would probably be a lot of work to make it work in my unusual shell. Probably the easiest output format to deal with (or read) is 'direnv export systemd'.

PPS: When setting VIRTUAL_ENV in your .envrc, remember to set it as 'export VIRTUAL_ENV=...' because it probably doesn't already exist in the environment. As the direnv documentation says in passing, your .envrc is loaded into a Bash subshell, so it can contain any valid Bash stuff and it needs to explicitly export (new) environment variables you want direnv to pick up.

(2 comments.)

Suppose, not hypothetically, that there's some technology that is having terrible effects on the world; it's practically destroying websites, it's worsening energy and water usage, and so on. Suppose also that there are real doubts about how good the technology is and what people will pay for it (since we mostly exist in a monetary economy and the latter is a consideration).

If you don't like this technology and would like to see it go down in flames (cf, also), there's an important difference between whether the technology is ineffective (it doesn't deliver particularly useful results once you peel off the hype) or if it's merely uneconomical. To put it bluntly, technologies that are merely uneconomical are much harder to get rid of than technologies that are ineffective. Obviously, if there's a useful but uneconomical technology, a bunch of people are going to try to make it economical and sooner or later they may succeed. Less obviously, there can be parties that care much less about the economics if they have something that's useful to them.

(Which is to say in part that governments are willing to spend lots of money on 'weapons', broadly defined. Many years ago I read someone say that computer security attacks that were merely expensive were actually attractive to government intelligence agencies; they feed the agency's budget and keep less well funded adversaries out. It's stuck with me ever since.)

Unfortunately for me and many other people, one particular such technology does seem to have reached the point where it's not ineffective in certain areas (cf). Worse, some of these areas are areas of deep interest to multiple governments, and some of those governments have a demonstrated ability to sustain very expensive, multi-year technology build outs for things that were uneconomical at the time. How unfortunate this development is will depend on how many areas the technology is reasonably effective at and the ultimate economics of those areas.

(Also, obviously, that a technology can be effective doesn't mean that the industry around it isn't a bubble. Bubbles are about the economics, not the underlying technology; it's entirely possible to have major bubbles with well proven technology, like home construction and renovation. Since I would like to replace my home desktop before it's ten years old for a reasonable amount of money, I do hope that current RAM and disk drive prices are due to a bubble and the bubble goes away before too long.)

That a technology is effective (in some area) and currently economical doesn't mean that you necessarily have to use it in that area, at least for personal things. You might choose to prioritize ethical and other concerns over "productivity". But unfortunately I'm less and less convinced that people in the technology field really care about such things, either in their paid work or outside of it.

(Corporations certainly don't care. "Externalities" are other people's problems unless and until something forces the corporation to care.)

PS: See also Iris Meredith's There's no such thing as Just a Tool. Technology is never neutral.

(One comment.)

We have a bunch of servers, which means that we need to give them (host)names. Like many places, we've had a number of naming schemes over the years; for example, many years ago the naming scheme was that servers were named after major north-south streets in Toronto (resulting in names like 'yonge' and 'bay') and other machines were named after east-west streets. However, over time we've gradually moved to mostly using boring and functional server names for many of them, names like 'comps1' and 'apps0'.

The history of the names of our fileservers provides a useful illustration of why we've wound up making this choice. They've always had names that started with 'san' (even in the recent generations where there's no SAN, cf), and for several generations these were city names like 'santafe', 'sanjavier', and 'sanandreas' (the test fileserver). However, we've never put these names in our NFS mounts; instead NFS filesystems are mounted from names like 'fs1' and 'fs3'. In the very old days this was because we could fail over these names (and their filesystems) from one physical NFS server to another, and then we carried on the habit.

However, this disconnect between fileserver host names and what they corresponded to created little frictions in practice. If you wanted to do something with the fileserver that hosted fs1 and its filesystems, you had to remember which san<whatever> name that was (in the current fileserver generation, because every generation needed a new set of host names). And if we received an email message about 'sanjavier', we might have to remember which fsN (virtual) fileserver that was. It wasn't impossible to deal with, but it was a bit annoying. So in the current generation of fileservers, we (mostly my co-workers, to be honest) decided that our fileservers would have boring names that directly mapped to their virtual fsN fileserver identities, so now they have host names like 'san1' and 'san4'. The result is less interesting host names but it's clearly easier to deal with on a day to day basis in practice.

(Being easier to deal with in practice is why our general multi-user servers were renamed from north-south streets to things like 'apps0' and 'comps0' when we moved from Solaris to Linux. We felt this would make it easier for normal people to remember and pick out what sort of machine they were logging in to, especially since we had a difference between general purpose login servers and compute servers, and people weren't supposed to run compute intensive things on the login servers.)

We do still have some interesting server hostnames, but they're generally for machines that normal people don't have to deal with and it's easy for us to keep straight, and some of them are sort of historical at this point. Our external mail gateway has been called 'alkyone' for roughly twenty years, but the current companion spam filtering server is called 'rspamd', after the primary software it runs, rather than 'keyx' (the old name of the server that ran our previous generation of anti-spam software). Our IMAP server is called 'aviary' (because it runs Dovecot), but we only have one and it's easy to keep straight.

(One of my co-workers has a Classics background and is great for classical Greek names, and another one likes Norse mythology. They come up with the interesting server names these days, because my naming sense is relatively bad and I know it. If I need to give a new server an interesting name, I ask them, and that's why a special DNS resolver is called 'argos'. The best my natural naming sense comes up with is things like 'sponge' for the name of our temporary backup MX server (because it was going to soak up incoming email temporarily).)

(5 comments.)

Suppose, not entirely hypothetically, that you'd like to block some people's logins to some server based on things other than what OpenSSH's sshd directly supports in its sshd_config, such as the login's shell. When I discussed this recently in a sidebar to this entry, I said that you could do this in PAM in either the 'session' or the 'authentication' (auth) PAM stack. I have to take that back, because I'm only half right.

The problem with blocking SSH logins using the PAM 'auth' stack is that OpenSSH's public key authentication is done outside of PAM. If a connection is authenticated through public key stuff, sshd never invokes the PAM auth stack and so your 'auth' based block never has a chance to take effect (this also limits some combination of authentication). However, there is an alternative in the 'account' PAM stack, which is specifically documented in pam.conf(5) as being useful for this sort of access control.

(I haven't previously had to deal with the 'account' PAM stack but I'm familiar with the 'auth' stack, which is why I reflexively reached for it as an access control mechanism. As we can see here, that's not necessarily a good idea in PAM. Hopefully I'll remember this for the next time.)

In my original entry, I didn't know how to provide a message for people trying to log in except through pam_motd, which only takes effect in the "session" PAM stack. On the Fediverse I was pointed to the useful pam_echo module, which will print messages in all PAM modes and take its message from a file (and the message can have some things dynamically expanded in it). If you need a dynamic message (for instance, if you have a number of different shells that produce different messages), the other option is to use pam_exec with its 'stdout' option to run a script that prints out a suitable message (which is then passed back through sshd and the person's ssh client). Pam_exec is started with a very minimal environment but you do get $PAM_USER, so you can look up their shell in /etc/passwd or otherwise figure out what message to generate.

Using the same convention as my first entry, this gives us a PAM stanza that looks like this:

account [success=ignore default=2] pam_succeed_if.so shell =~ /admin/shells/*
account optional  pam_echo.so file=/admin/access-denied.txt
# or
#account optional pam_exec.so stdout /admin/access-denied.sh
account requisite pam_deny.so

We use the PAM control value 'requisite' instead of the more commonly seen 'required' so that the account PAM stack fails immediately (and see my entry on understanding the effect of PAM module results, and there are some issues if this is a substack).

The pam_succeed_if module has a relatively good set of data fields and comparisons it can do that's broadly more generous than what sshd_config provides in Match statements, so you may wind up wanting (or needing) to use it instead of sshd's native mechanisms. If you're sure you want your access block to apply only to SSH logins as opposed to everything, you can also put this stuff in a suitable place in /etc/pam.d/sshd instead of a more general file.

(In our case, if we give someone an administrative shell, we want any lingering other things that might run programs as them to also not work.)

As I mentioned in yesterday's entry, the standard systemd behavior is that anyone can do 'loginctl enable-linger' for themselves. We don't want people to do this on our servers, partly because it doesn't work the way people expect in our NFS fileserver environment and partly for other reasons. Today I worked out how to disable this, and refreshed my memory of Polkit (also) in the process (Polkit is how systemd controls authorization for many of its operations that normal people can theoretically invoke).

The default requirements for loginctl actions are set in a .policy file in /usr/share/polkit-1/actions, specifically in org.freedesktop.login1.policy (because it's systemd's logind that is really doing the work). If you look in this XML, you'll find there are two relevant Polkit actions:

[...]
<action id="org.freedesktop.login1.set-self-linger">
[...]
<action id="org.freedesktop.login1.set-user-linger">
[...]

(It's conventional that most of the Polkit action name is from the D-Bus service doing things, in this case org.freedesktop.login1.)

The first one is for 'enable-linger' or 'disable-linger' on yourself, and the second is for doing it on other people. The initial Polkit policy, set in the '<default>' block of each <action> block, is to allow it for yourself and require administrator authentication for other people. Unfortunately org.freedesktop.login1 doesn't distinguish between enabling linger on yourself and disabling linger on yourself; both are covered by the same action and the same set of permissions.

(Loginctl is clever enough to use set-self-linger even if you give it your own login name as the command line argument.)

The pkaction command can be a more convenient and readable way to get this information, but I didn't look at it until after I'd found the .policy file.

To change the initial, default behavior we need a Polkit rule in a .rules file in /etc/polkit-1/rules.d. The contents of this file are fairly straightforward if you've (re)read other rules files to refresh your memory:

polkit.addRule(function(action, subject) {
    if (action.id == "org.freedesktop.login1.set-self-linger") {
        return polkit.Result.NO;
    }
});

Since we unconditionally return NO for all attempts to use this action, all non-root requests for it are rejected. People who try will get a relatively terse message of "Could not enable linger: Access denied" (loginctl says 'enable' even if you're doing disable-linger).

As I discovered with some experimentation, if you're root (UID 0), this is ignored and you can still do 'loginctl enable-linger' (in addition to 'loginctl enable-linger somelogin'). However, if you're su'd to root, doing 'loginctl enable-linger' enables lingering for your own login, not 'root', presumably because that's who systemd considers you to be at the moment.

Sidebar: 'linger' and initially inaccessible home directories

In any environment where people's home directories aren't immediately accessible as the system boots, enabling 'linger' status for a login is not merely ineffective, it's actually damaging. Systemd starts the user manager for 'linger' users during system boot, and if someone's home directory isn't present at that point, their newly started user manager won't see any special user unit files they have set up. Later, when their home directory is available and they log in, their user manager will already have started so it doesn't notice the now available $HOME/.config/systemd/user/default.target.wants and trigger unit startup.

You can manually start units once your $HOME is available and you log in. But as far as I can see, nothing will auto-start them for you, not even using systemctl to reload or re-exec your user manager. To get everything auto-started properly, you need to get your user manager to exit and then log in again so it's restarted from scratch, for example by doing 'loginctl terminate-user ""'. The empty double quotes are necessary; you must give loginctl an explicit empty argument as the login to terminate, presumably for safety reasons.

(Terminating another user is Polkit authenticated as one of the things included in 'org.freedesktop.login1.manage'. Terminating yourself doesn't seem to be authenticated at all through Polkit; logind just allows it.)

In our fileserver environment, people's NFS based home directories aren't available sufficiently early in boot to be there when systemd starts user managers for 'linger' logins. I'm not quite sure why this is so, because the service unit that mounts them (using our 'automounter' system) is WantedBy remote-fs.target. But the result is that setting your login to 'linger' is a subtle trap (and also unnecessary; you can keep your user manager alive in other ways).

(One comment.)

Suppose, not hypothetically, that you operate a multi-server Linux environment where most people get to log in to some machines, such as your general purpose login servers, but not log in to others that still have to carry your full set of logins, such as your fileservers or your SLURM nodes (people submit jobs through SLURM but aren't supposed to directly SSH in). The only good way to implement this today that I know of is to modify people's shell in /etc/passwd. Although this is flawed in general, most of those flaws don't apply if what you want to do is specifically stop people SSH'ing in to some machines.

(Let's assume you can't use /etc/nologin or the like because some people do need to log in, such as system staff.)

Now suppose that some people are hip to various systemd features or are just following setup guides, and so are setting up per-user systemd service units (also). Per-user systemd service units that are added by people are normally located in $HOME/.config/systemd/user, and if set up as WantedBy default.target, they're started by your systemd user manager when it starts, which is normally approximately when you log in. It turns out that user unit startup happens automatically regardless of what your $SHELL is, even if it's a do-nothing shell that immediately exits.

If a person with user units set up SSH's to one of your no-login servers where they have a custom /etc/passwd shell that tells them about this and then exits, systemd will start a user manager before it gets anywhere near trying to run their shell. This user manager will start their user units, which results in them running programs on a machine you don't want them to. If you have shared NFS home directories, this can happen entirely innocently.

(It's perfectly reasonable for someone to set up their systemd units with no specific host restrictions on what host they run on. If they only normally log in to one host they might not notice for some time until the day they, say, accidentally SSH to a SLURM node.)

This doesn't necessarily leave those user service units and their programs running forever. Under normal circumstances, your systemd user manager sits around until all sessions and .scope units under it have exited, waits another UserStopDelaySec seconds (normally ten seconds), and then shuts down all remaining units and exits. However, anyone can enable 'user lingering' for themselves via 'loginctl enable-linger ...', at which point their user manager and their service units are perpetual (who has linger status enabled is visible in /var/lib/systemd/linger, cf). A person who wanted to actively exploit this could run loginctl from a script in the ten second window they have after their initial SSH connection.

(There is a separate logind.conf setting, KillUserProcesses, but this only applies to processes in the session's .scope unit, not to processes in separate user service units, and almost no one turns it on because of the carnage that would ensue. Ubuntu 26.04 LTS definitely doesn't.)

As far as I know there's no easy way to get systemd to not start systemd user units if your shell isn't a standard one. For your own personal units, you can make them conditional on $SHELL with something like 'ConditionEnvironment=SHELL=/bin/bash', but of course you have to know about this and care. System wide, your administrative shells can do 'loginctl terminate-user ""' at the end, after printing their message, which will theoretically terminate the person's entire user manager; however, I think this can be incomplete under some circumstances. The normal OpenSSH sshd will refuse to let you log in at all if your /etc/passwd shell doesn't exist, but this may have side effects if other programs (such as your IMAP server) also fail authentication in this case.

In systemd v258 and later (which covers Ubuntu 26.04), you can force systemd to not create a user manager for people by setting a pam_systemd option of 'class=user-light' for SSH logins (or all logins if people only log in via SSH); SSH logins normally wind up with 'class=user'. For Ubuntu 26.04, a suitable PAM incantations for this might be:

session [success=ignore default=2] pam_succeed_if.so shell =~ /admin/shells/*
session optional  pam_systemd.so class=user-light
session [success=1 default=ignore] pam_succeed_if.so shell =~ /admin/shells/*
session optional  pam_systemd.so

(Yes, this is arcane. I wouldn't make this conditional on the service being ssh(d), because you probably don't want any session to start people's user units. Adding 'quiet' to the pam_succeed_if.so options may be useful.)

The advantage of this PAM approach is that people will still be put into the proper systemd unit hierarchy (for the possibly brief duration of the administrative shell), with a user-NNN.slice and session-NNN.scope unit. This may make other bits of your environment happier with life (for instance if you have scripts that trigger on login to apply per-person limits to their user-NNN.slice systemd unit).

(This elaborates on some Fediverse posts of mine.)

PS: loginctl's Polkit policy specifically permit everyone to set their own linger status, not subject to any particular Polkit rules. I've forgotten my Polkit knowledge since back then so I'm not sure if you can override this with your own Polkit rules. I may wind up finding out.

Sidebar: Rejecting logins, the other PAM approach

You can use PAM to selectively reject logins using pam_succeed_if.so to check the shell, and then possibly pam_deny.so to shut the PAM process down. There are two places in the PAM process where you can do this, in the authentication stack and in the session stack. Doing this in the authentication stack is cleanest (you're refusing to authenticate the login because it has a bad shell), but this has the drawback that you can't provide people any sort of message.

If you refuse in the session stack, you can use pam_motd.so to also provide a message, for example:

session [success=ignore default=3] pam_succeed_if.so shell =~ /admin/shells/*
session optional pam_motd.so motd=/admin/access-denied.txt
session required pam_deny.so

(You have to put this before the pam_systemd.so module for obvious reasons.)

This will show /admin/access-denied.txt and then drop people off. One drawback is that on Ubuntu 26.04 it seems to trigger some additional log messages:

sshd-session: error: PAM: pam_open_session(): Cannot make/remove an entry for the specified session
sshd-session: syslogin_perform_logout: logout() returned an error

This feels unsurprising; sshd and PAM together opened a session only to have the session yanked out from underneath them.

The other drawback is that it's going to get increasingly annoying if you want to provide different messages for different administrative shells (the more shells, the more annoying). Each different message will need its own three-line block in your PAM session stack unless you're lucky.

(2 comments.)

On the Fediverse, I said:

It has been '0' days since I discovered that I shot myself in the foot in Python 3 with a bytes/Unicode confusion issue, although this one was an unusually creative and subtle way to get things wrong (ie, it had no exceptions, it generated corrupted text).

This will require some background. I handle my email in the venerable (N)MH, which is very old and thus slightly flawed. One of those flaws is that while NMH can quote a true plain text message that you're replying to, you need to set up your own, additional stuff to properly quote MIME-encoded email in replies. Specifically, you need a program that will MIME-decode and pretty-print every text-based body part you want to quote. I have a long standing Python program for this, and not too long ago I moved it from Python 2 to Python 3 (which involved some learning experiences).

Recently, a number of emails I replied to started showing up with Unicode characters that rendered as a reverse video '?' glyph scattered at various random places in the quoted body. Investigation revealed that these were U+FFFF characters, but I didn't see anything odd in the message I was replying to (and thus quoting). Only after some digging did I work out what was going on and how I'd shot myself in the foot.

My MIME body part decoder gets passed the MIME Content-Type and Content-Transfer-Encoding on the command line and receives the raw text of the MIME body on standard input (this is basically required by how NMH works). To decode and parse the MIME body part, I was doing the obvious thing; I constructed an email.parser.Parser, read in standard input to get the body of the MIME part, put together a minimal set of MIME headers and the body, and passed the result to Parser.parsestr(). Some of you may already see my problem.

The messages I was replying to had a content-type that said they were UTF-8 ('text/plain; charset=utf-8') in a content transfer encoding of '8bit', and they contained Unicode characters (codepoints) in the range U+0080 through U+00FF (in this case, my old nemesis U+00A0). Because this message part used '8bit' transfer encoding, these Unicode characters were directly represented as UTF-8 bytes in the raw MIME body part passed to my program on standard input; when I read them directly from sys.stdin as text, Python 3 did what I'd asked for and decoded stdin as UTF-8, recreating in my decoded Unicode string those U+00A0 characters. Then I passed what I claimed was a UTF-8 encoded thing to Parser.parsestr() and it dutifully tried to decode UTF-8 again (effectively interpreting the Unicode codepoints in my string as bytes), found some invalid UTF-8 sequences in the form of those U+00A0 codepoints, and put in a U+FFFF marker for each of them.

(If you do this trick with Unicode codepoints that are above U+00FF, Parser.parsestr() will eventually give you back '\u20ac' style escapes, which is at least more obvious about what's gone wrong.)

The solution (and correct answer) was to switch to email.parser.BytesParser and its parsebytes() method, with a standard input that's been read in raw mode from sys.stdin.buffer. This aligns everything properly so that I'm handing raw text to the thing that decodes raw text, rather than somewhat cooked text.

(This issue specifically required that the content transfer encoding be '8bit', not '7bit'. Had the MIME part been encoded down to 7-bit clean as '7bit', there wouldn't have been any straight UTF-8 in the input for me (well, sys.stdin) to decode to Unicode codepoints. This is probably why I didn't notice the problem immediately after the Python 3 conversion.)

Sidebar: Tracing the code

If you use email.parser.Parser with a Unicode string in my bug situation, the resulting email.message.EmailMessage object will hold your Unicode string as its ._payload private attribute. When you call its .get_content(), the policy system will wind up calling back to .get_payload(decode=True). This will wind up encoding your Unicode string to a bytestring with (in my case) 'str.encode("raw-unicode-escape")', which turns Unicode over U+00FF into the \uNNNN form but leaves U+0080 through U+00FF as single bytes. This bytestring is then run through 'bytes.decode("utf-8", errors="replace")', which is what replaces the \xa0 byte with U+FFFF.

See email/contentmanager.py and email/message.py, also, all of which I'm noting down here because I don't want to have to research this a third time.

Suppose, not hypothetically, that you want to use systemd memory controls on either system.slice or user.slice to keep people from running the system out of memory. In either case, you need to come up with some amount of memory that you'll reserve for system.slice (and the kernel), through either setting MemoryMin= on system.slice or setting MemoryMax= on user.slice.

If you're setting a maximum, you need to know how much memory the system has. As covered in proc_meminfo(5), /proc/meminfo's MemTotal field gives you the total theoretically available system RAM after reserved bits and the kernel's very basic code and other memory usage. In practice the kernel is likely to need more RAM than this in normal operation, and figuring out where the memory is going can be complicated. Also, it's likely that in order to perform well, everything under system.slice will need more memory than simply the total program memory usage, because cgroup and thus systemd memory limits include things like the kernel filesystem cache.

(My case of too-small CPU and RAM limits significantly harming program performance was definitely partly from having too little RAM, probably partly by hampering the filesystem cache.)

At the moment, system.slice on the servers I care about for this appears to be using about 1.6 GB of user level RAM and anywhere up to just over 2 GB of kernel slab cache. Unfortunately I don't have metrics over time for this (yes, I'm now tempted to build something to collect them). Filesystem cache usage is all over the map, but I should probably allow at least 1 GB for that, and more might be better. I could round this up and say we should reserve 5 GB for the 'system', broadly defined, and set MemoryMax= on user.slice accordingly (I prefer the approach of capping user.slice's memory usage to trying to set a minimum memory for system.slice).

(This turns out to actually be typical for system.slice across most of our machines, with less active machines having under 1G and some under 512M. The kernel file cache size for system.slice varies hugely, of course, and some of our machines are exceptions because they run heavyweight things as system .service units, such as our metrics system.)

I would need a script to dynamically compute this size from MemTotal in /proc/meminfo, but that's not particularly difficult (especially since I have a script to copy chunks from; it's used to size the ZFS ARC on our ZFS fileservers based on the amount of memory the fileserver has). Our existing system for setting up fair-share CPU scheduling between people runs at login time, when we know a user.slice exists (because a user-<uid>.slice for the person who's logging in now exists as its child), so we can compute the right MemoryMax value and set it on user.slice. There's even a convenient place to put this in our current script framework.

However, this is definitely the sort of change that I should monitor after it's made to make sure that I'm correct about how much memory everything actually needs and uses (cf). That means I need some basic monitoring of top level systemd cgroup memory usage that feeds into our metrics system, which I'll probably do with a minimal shell script rather than anything more elaborate.

(There's at least one Prometheus systemd exporter that provides metrics for systemd units, but the last time I looked there was nothing that provided this kind of memory usage information.)

For reasons beyond the scope of this entry, I've been manually maintaining a group of files in sync across three 'machines' (my GNU Emacs configuration). There is no master machine for edits (it depends on what I'm on at the time when I need a change), and I don't always propagate changes immediately after I make them. Plus, some of these files are in various places under my home directory. I made a lot of use of 'ssh <host> cat <relative path>/file | diff -u - file' and the like to keep track of this all and make sure I wasn't overwriting changes when I propagated things, and I kept doing it by hand for various reasons.

(I'm aware that some people would solve this problem with things like a Git repository. This isn't my way for various reasons.)

Today I sat down and wrote remdiff and remsync scripts to do this for me. These scripts are a little bit more elaborate than they need to be (partly because I put in a reasonable amount of error checking), but they still weren't much work and they're not very long. Getting a reasonably decent design for the command line arguments took a few iterations of writing code and then using the scripts and being unhappy, and I'm probably going to fiddle more, but the result is usable already.

The existence of these scripts doesn't just reduce the amount of effort and irritation involved in this process. Because they make it easier to do this, they encourage me to do it more often and thus reduce the chances I will have pending changes on more than one machine that I'll have to reconcile. They also make it feasible to bulk check a whole collection of files, which found some differences I hadn't realized.

Writing shell scripts is one of those Unix superpowers. Unix has such a collection of reusable tools and tool components (plus various shell features that are designed to make scripting easier) that it's practically a shame not to use them. But in this case and others, I'll tell myself over and over that it's a bit too trivial or too much work or whatever to turn something into a shell script. So today I was once again reminded that I should write more shell scripts than I do.

Another non-obvious advantage of shell scripts is that you can adjust their command line arguments to exploit shell features like filename completion. In the original manual version, I couldn't use shell filename completion for the '<relative-path>/file' bit because I was already in the <relative-path> directory on my machine. Since I no longer have to supply this argument by hand, I can do things like 'remdiff host: fly<TAB>' to have my shell auto-complete the filename to 'fly-startup.el'.

(People who use fancy shells could arrange for their shell to know how to autocomplete hostnames as well, bringing this down to, say, 'remd<TAB> ho<TAB> fly<TAB>'. For this sort of personal command you could adopt a very brute force approach based on knowing what hostnames you're likely to ever use it on. If you want to use another host, you add it manually to your completion information for this command.)

PS: It figures that the moment I had 'remdiff' I started using it to check other files that I generally keep in sync between multiple systems. Apparently this was yet another one of those little points of friction that I hadn't realized I had.

(One reason I held back on writing 'remdiff', which was the first of the two scripts, was that I expected parts of it to be an annoying pain to write, like going from the absolute path of the current directory to its relative path from $HOME. And I was right, bits of the script do feel like an annoying pain, but it's worth it.)

Sidebar: My brute force approach to getting the relative path

In Bash:

HDIR=$(realpath "$HOME")
MPWD=$(pwd)
# We could be in $HOME.
if [ "$HDIR" = "$MPWD" ]; then
    RELDIR="."
else
    RELDIR=${MPWD##"$HDIR"/}
fi

(Using '##' this way is actually a standard Bourne shell feature now that I look.)

As far as I know there's no command that will do the '${..##../}' bit to strip off some but not all leading directory components on a path. I need the realpath here due to a long standing local feature that causes my $HOME to have a symlink in it.

(I've decided it's a feature that I'm not explicitly restricting this to be under $HOME; if I'm outside of $HOME, it will use the same absolute path on the local and remote machines. This turns out to be convenient for some things.)

(3 comments.)

One of the things that ruff can be is a Python linter (in addition to at least a formatter and a not very attractive language server). In its default configuration, ruff is a quite opinionated linter with plenty of views that I disagree with, but it also has some useful suggestions for modernizing code that's been converted from Python 2, which I have a lot of and would like help in fixing up. Fortunately it's possible to change ruff's configuration so that it skips complaining about views that I disagree with, both globally and on a per-project basis.

(One very frequent wish ruff has is for me to use format specifiers (f-strings) rather than % format things (cf). I decline.)

On a normal Linux system, ruff's global configuration file is ~/.config/ruff/pyproject.toml, as covered in ruff's configuration documentation. To ignore specific ruff warning codes, I want something like this:

[tool.ruff]
lint.ignore = ["I001", "UP031"]

(Ruff can also ignore entire classes of rules but I haven't gone that far yet.)

I've found it useful to write comments describing what each error code is, so later I can figure out what "I001" is without having to go through ruff's list of rules and perhaps its current default rules, which change from time to time (although default rules are also marked in the list of rules). I typically find rules to add by running ruff (either directly through 'ruff check' or through GNU Emacs) and then seeing the rule number of things it's complaining about that I disagree with.

(In writing this entry I found that E401 is no longer a default rule, for example, so my global setting to ignore it isn't necessary any more.)

Ruff also has per-'project' configuration, which in my view is most conveniently put in a pyproject.toml file at the top level of your project (rather than, say, the ruff specific ruff.toml, cf)). This per-project file can extend your global file if you want:

[tool.ruff]
extend="/u/cks/.config/ruff/pyproject.toml"
target-version="py311"

[tool.ruff.lint]
ignore = ["UP018"]
extend-safe-fixes = ["SIM118", "RUF059"]

If you use 'extend' to pull in your global default settings, your 'ignore' list is added to the global ignore list, as documented (well, sort of, see extend-ignore). If you want to re-enable some rules that you disabled in your global configuration, I think you have to copy your global pyproject.toml into the project, modify it as appropriate, and not use 'extend' at all.

Explicitly extending what ruff lint fixes are considered 'safe' is useful for persuading some external systems, such as current versions of Flycheck in GNU Emacs, to recognize them as available fixes. For whatever reasons, some systems don't recognize unsafe fixes even when you enable ruff offering them.

Mypy has similar global and local configuration control. As covered in The mypy configuration file, the global location is ~/.config/mypy/config. In my view, the local configuration is best also put into a pyproject.toml file, where you might disable specific errors like this:

[tool.mypy]
disable_error_code = [ "var-annotated", "attr-defined" ]

Your tool of choice for showing mypy messages may or may not report the actual error code in addition to the messages, so periodically I get to consult mypy's Error codes documentation and also. This documentation also covers inline comments to disable them on specific lines.

Both mypy and ruff have inline configuration, per mypy's inline documentation (also) and ruff's linter error suppression documentation. For ruff, I prefer the line-based '# ruff: ignore[...]' syntax over the other options. For mypy, I can possibly work around typing complaints rather than suppressing them.

Today I use an extensively customized X environment, and I have every intention of clinging grimly to it for as long as I can. But Wayland is the future, even if the future isn't arriving very fast, and someday I'll have to move to Wayland. For a long time I've expected that to be very unpleasant, with a lot of differences and a significantly inferior environment as the result. However, these days I'm somewhat more optimistic, although I still expect a lot of pain (and to have to rebuild or replace a lot of programs).

This optimism is partly a product of finding out things like the Arch wiki's list of input remapping utilities, because I'll need a replacement for xcape, and their section on application launchers (for a dmenu replacement). But it's also because there are some real moves to bring a more X like experience to Wayland. River separates Wayland compositing from window management, making it much easier to create window managers, and one of the people involved with fvwm has used River to build cow, a fvwm-like window manager. Cow is especially interesting to me because it has a version of FvwmIconMan, which is a cornerstone of my desktop.

Creating something like my current desktop in Wayland today would still be a lot of work (and it's artificial make-work, since I'd wind up no better off than today). I'd certainly keep on using a lot of X programs via rootless XWayland and I'd have to learn things like waypipe to handle remote Wayland-only programs. But it at least seems possible now, when before I assumed that if I had to move to Wayland, I'd wind up in something that looked more like Cinnamon (which is basically the lowest denominator of desktop experiences I can stand; modern Gnome is not for me).

Of course the optimistic hope is that non-Linux platforms like FreeBSD, OpenBSD, and NetBSD can apply enough pressure to keep GTK and other toolkits working on X, and crucial software like Firefox doesn't give up on X compatibility. Then it's likely that I can keep on using X for years and years to come (I hope for decades). Maybe I'd even wind up with a reverse Wayland to X bridge to let Wayland programs run (rootless) on my X server, should I need to use some Wayland-only stuff.

(And in other reasons for optimism, the Xorg X server is working on making its first official release in a while.)

PS: My understanding is that while Gnome and KDE are dropping their X session support, it's expected that Gnome and KDE programs will keep working on X. If Gnome and KDE programs drop X support, well, I guess I'd get to see how many of them I really need. I'd certainly miss some of them, ironically including gnome-terminal these days.

(5 comments.)

We customize and set up a lot of things, sometimes on a per-system basis. Because university departments can be long-term places and we have our own eccentric setup for configuration management, these settings can persist for a long time, across multiple Ubuntu LTS versions and so on. Sometimes the result outright doesn't work at all, with consequences that range from undesirable to mostly harmless (for example, if you're trying to set Linux kernel sysctls that no longer exist). But sometimes our old settings are actively harmful, as happened recently with the per-user limits we set on everyone on one machine. As of Ubuntu 26.04, these limits are now visibly too small and cause undesired behavior.

We've used these particular settings for a a while, probably since mid 2022; back then they didn't cause problems (or at least, none that we noticed). But a lot changes in Linux over four years, especially since this particular server was running Ubuntu 18.04 at the time. However, every time we rebuilt it with a new Ubuntu version (and new hardware), we simply carried over the 2022 settings, because that was the easiest way. We never paused to check if the settings still did what we wanted without undesired side effects; if nothing obvious blew up we assumed it was all good.

(Partly this is an artifact of how we build systems. Effectively our process is 'install some version of Ubuntu and then layer these changes over top'. When updating what could be called the base Ubuntu version, there's nothing that pushes us to rebuild our changes from scratch, reconsidering each one to see if it's still right; instead, it's easy and obvious to carry over the set of changes for the previous version.)

It's possible we have other carried-over local settings that are now harmful, and it's probable that we have some that are unnecessary or ineffectual (although I think I checked that all of the sysctls we set are still there in Ubuntu 26.04). Getting rid of any settings that are now harmful is obviously a good idea, and cleaning up stuff that doesn't work any more is good in general to reduce complexity and confusion. The question is how.

I don't know if it's sensible to try to re-check every setting we make every time we switch to a new Ubuntu version, given that we have a bunch of settings and they're usually fine and still what we want. But we (I) probably should plan to check them once in a while, rather than waiting for things to blow up. One possible answer is that now that one setting has blown up in 26.04, I should take this as a sign and audit them all, but I'm not sure I have either the enthusiasm or the time for it (any more than I had time and enthusiasm to investigate some ideas).

('Re-check your settings every Ubuntu LTS release' is sort of like 'floss your teeth after every meal' in that it's perfectly correct and also not something most people are actually going to do.)

Still, it's good to be reminded periodically that this can happen (and to us). If nothing else, maybe I'll be more suspicious of old settings the next time something unusual happens.

(3 comments.)

Yesterday I wrote about how on one of our servers, systemd took three seconds to load (user) units as part of setting up your systemd user session. I also mentioned that various operations were unusually slow on the system in general, but I couldn't see anything in our metrics system to suggest slow IO or the like. Today, I solved both problems and it turned out to be our own fault.

This particular server is our SLURM cluster's 'head node' (master server), which we allow people to log in to because it used to be the only place you could submit SLURM jobs from. However, we had long standing problems with people accidentally running their CPU-consuming jobs on this SLURM head node instead of in a submitted SLURM job, or doing CPU and memory-intensive preparation work on it (including personal package installs, which in some languages and for some sets of packages can be quite resource intensive). In order to discourage this, we wound up setting very low systemd-based CPU and memory limits on people's user-<uid>.slice units (cf); for years, the limits were 128 MBytes of RAM and 25% of (one) CPU. It turns out that in Ubuntu 26.04, these limits are sufficiently low that things such as 'dpkg -S' run appreciably slowly.

(I don't know if it's the memory limit, the CPU limit, or the kernel impact of setting a CPU limit below full time use of one CPU. This didn't used to happen on previous Ubuntu versions, at least not to such a noticeable degree, but programs keep growing and so on. Other login servers with per-user limits have much bigger limits and so don't run into this even on 26.04.)

That turned out to be the cause of all of the slowness I saw once I was logged in (in dpkg and other things), but it didn't seem to explain the slowness in systemd user session setup, because these limits are only set after the systemd user session has been created (we set them through pam_exec running a script after pam_systemd has run and made the systemd user session if necessary). And, famously, your user-<uid>.slice unit goes away after you entirely log out, so the old limit settings weren't being carried over through an old user-<uid>.slice unit. Or so I thought, but I was wrong.

Our per-user limits are set with 'systemctl --runtime set-property user-<uid>.slice ...'. As documented, --runtime doesn't just change the running unit in memory, it also writes your changes to files in /run/systemd/system.control/<unit>.d/, and these written out files are then persistent until the next reboot wipes out /run. When systemd recreated a user-<uid>.slice that had already been set up with limits (for example, because I'd logged in once already), those /run files meant that it re-applied our (low) limits right from the start and then the systemd user session process was constrained and (drastically) slowed by those limits. Raising the limits to more reasonable values completely eliminated the slow systemd user session startup, making systemd's user session startup as fast as expected.

I had vaguely known that --runtime wrote things to /run to persist them, but I hadn't put the pieces together in my mind to realize that the per-user 'user-<uid>.slice' units would immediately have those limits applied again the moment they started. So I believed that all systemd user sessions started out completely unlimited, even if I'd logged in before, and limits were only applied once the session existed. A consequence of this that I want to remember is that stopping setting limits on someone at login time doesn't remove limits existing limits set in past login sessions, even if the person logged out completely. To undo setting per-user limits, you need to clear them somehow (or reboot the machine).

(Systemd may cache things in memory, so I'm not sure if wiping out the /run files is sufficient to completely reset the situation. I believe clearing CPU limits in cgroup v2 is done by setting 'CPUQuotaPerSecUSec' to 'infinity' and clearing memory limits is done by setting 'MemoryMax' and/or 'MemoryHigh' to 'infinity'. Set 'TaskMax' to some large number; the system default varies from machine to machine.)

I'll start with what I said on the Fediverse a while back:

Today's mystery: on exactly one system (with shared NFS home directories), logins take 3+ seconds to start because '/usr/lib/systemd/systemd --user' sits on its hands between running systemd-xdg-autostart-generator and running the next step of a remarkably opaque login process.

I don't even know where to start and I only got this far with a hacked version of the great extrace that also reports a process start time number.

(Although I didn't mention it in the post, these are SSH logins.)

As mentioned, this only happens on one system (running Ubuntu 26.04). Other systems, with the same software configuration and the same NFS mounted home directories, don't have anything like this slow login. The slowness has persisted over multiple reboots of the system (and a hostname change as it moved into production). Once a systemd user session exists because of an ongoing SSH login, further SSH logins run much faster. As far as I can tell, this isn't specific to my own login, it happens to other ones as well.

(This leads to my hack workaround for my own account, 'ssh -o "ControlPersist 30d" -N -M <host> -f', which not only keeps my session active basically forever but also sets it up so I don't even need to start up a new SSH connection as such.)

You can extract a bunch of information about your session startup with 'systemd-analyze --user dump', and when I did this there was a clear problem:

Timestamp units-load-start: Thu 2026-07-09 18:40:30 EDT
Timestamp units-load-finish: Thu 2026-07-09 18:40:33 EDT

Three seconds to do this isn't normal, and this is before systemd starts really doing anything (but after unit generators have run).

This was traced to a specific section of the systemd code, cf, which makes only two calls, manager_enumerate_perpetual() and manager_enumerate() (which are also in manager.c). These appear to set up all of the known systemd units of various types (cf). There seem to only be a few perpetual units (especially for user sessions), things like '-.slice' and 'init.scope'. Non-perpetual enumeration appears to happen for devices, mounts, and swap, and then triggers processing the 'load queue' (of units).

Unfortunately, using opensnoop-bpfcc to watch what things get opened shows nothing obvious (although it does show significant gaps in file open activity). More detailed system call traces might provide some more insight, but I don't know of any good way to do system call tracing outside of strace, ideally narrowing the trace down to a process running under a specific UID with a specific command name (the top level systemd user session process is started on the fly, so its PID is unpredictable in advance).

At the root, I suspect that this is something other than systemd, and systemd is just accessing whatever is slow, or doing some operation a lot that is unusually slow on this system. There are some other odd indicators on this system, where other things are sometimes slower than I expect them to be (such as 'dpkg -S'), but at the same time I can't find any obvious signs in our metrics system.

(I've even considered thoughts like a throttled CPU or slow RAM, but I can't find any particular signs of that and it feels like I may be distracting myself with complex explanations for what may turn out to have a simple cause.)

(3 comments.)

Today I asked a question over on the Fediverse:

In GNU Emacs, is there any way to 'undo' the loss of a selection region, especially a rectangle selection? I am forever going through the dance of C-x SPC, select a rectangle region, do something that fumbles the action I wanted to do, rectangle region disappears, I grind my teeth, and I would like an easy way to recover should there be one.

(I can't spot anything in the manual or in MELPA/ELPA packages, but I may not have looked hard enough.)

Thanks to a very helpful reply and some additional experimentation, I found my answer. What I want to do under normal circumstances is C-x C-x, C-x C-x again, and then C-x SPC.

What I called a rectangle selection is more correctly called a region rectangle, a rectangular region. In GNU Emacs, a region in general is the text between point (the cursor) and an active mark. When I do the wrong thing with a region rectangle active, the mark and the region (rectangle) deactivate, but the mark remains. When I do the first C-x C-x (exchange-point-and-mark), point and mark flip and the mark is reactivated so I have an active region again, although not a region rectangle. However my cursor is now at the start of the region, not where it was when I fumbled things. Doing C-x C-x flips point and mark again, leaving the mark and region active and returning my cursor to where it was. Then C-x SPC (rectangle-mark-mode) switches from a regular region to a region rectangle.

(Until now I hadn't realized that C-x SPC was a toggle, not a stand-alone key binding that started a special rectangular region mode. Learning this was a useful revelation. If you never use it with a region already active, C-x SPC looks like a stand-alone thing because it immediately activates a mark at point if there isn't already an active mark and thus a region.)

With a region rectangle active, C-x C-x cycles between the four corners of the region (as documented, cf). If the region is zero characters wide (for example, because you're drawing it down the left side of a block of text so you can use 'C-x r t' to insert '> ' in front of every line), this is equivalent to just swapping between the start and the end, which is what C-x C-x does with a regular region active.

(You can also use string-insert-rectangle for this insertion of a prefix, especially since it defaults to '> ' as the string to insert. But 'C-x r t' on a zero-width rectangle fits my mental model better, since it's close to how I do it in Vim block editing, and also it actually shows me the effect of the text I'm inserting. And there's also comment-region, which in a text buffer will ask you what 'comment' string you want to use and which works with a regular, non-finicky region.)

In fact, even 'making' a region rectangle with C-x SPC isn't necessary for what I normally do (inserting text at the start of all lines), because now that I actually read the documentation (and do some experiments), the 'C-x r ...' collection of commands don't necessarily require an active region rectangle. C-x SPC will let you visualize the rectangle they'll act on, which is potentially important, but if I set a mark at the start of a paragraph (in the leftmost column) and then move to the end of the paragraph (or just after it) but still in the leftmost column, 'C-x r t' will do what I want. M-{ and M-} are obvious movement commands to use here.

(There are some commands that behave differently depending on a region rectangle and a regular region, like C-w. I think I'm going to keep using C-x SPC to make region rectangles before C-x r t, simply so I can keep things straight in my head.)

One of the lessons I draw from this is that I want to try to read Emacs documentation more carefully, rather than skim it. I obviously found C-x SPC in the Rectangles documentation, but I didn't read carefully enough to spot that it was a toggle (or think deeply enough about the convenient implications of that). Another is that I probably have a lot of Emacs semi-knowledge that has decayed to the level of superstition by now, such as my understanding of regions and the mark (until I refreshed it by reading The Mark and the Region documentation for this entry).

(Emacs is truly a voyage of discovery, in some ways much more so than Vim.)

PS: This means that if I accidentally deactivated a regular region, I can get it back with C-x C-x then C-x C-x again. Or just C-x C-x if I don't care about which end is point and which is mark, but usually I do (and I want mark to be the start and point the end). And in something else I want to remember, there's C-M mouse-1 to start sweeping out a region rectangle; after it covers something, I can switch to regular cursor movements for more precision.

Tramp is a famous and often praised GNU Emacs system for editing remote files; lots of people will call it one of Emacs' compelling features. As part of a big GNU Emacs renovation earlier this year, I made Tramp actually work for me, and at that point I expected that it would be a curiosity and I'd mostly not use it. That has half worked out the way I expected, and the other half is that I'm editing this Wandering Thoughts entry with Tramp, as I have most of my entries since then.

For various reasons, including that I now read my email mostly through GNU Emacs, I have a full GNU Emacs environment set up on our login servers, complete with the same custom build of GNU Emacs that I run on my desktops and the same Emacs Lisp. Under normal circumstances I'd rather use that Emacs (either with or without X forwarding) rather than wrestle with Tramp, because Tramp is getting me relatively little but is adding complexity (and it may be slower in practice to use a local Emacs with Tramp but a remote LSP server, because LSP servers are kind of chatty). There are some situations where I want to try using Tramp (for example, using compare-windows to compare the same file from two different machines, but they're uncommon and I haven't gotten around to it yet.

(Tramp can work with LSP servers, Git, and so on, but all of them are run remotely, which means if you have a remote Emacs too you have all of the pieces you need for a good remote Emacs environment. And there's stuff that's easy to make work locally but awkward over TRAMP.)

Tramp has a certain number of appealing additional features for system administrators specifically, such as using su and other programs to edit root-only files, especially over SSH. If you want to use a fully set up Emacs environment (ideally your regular one) but still edit root owned files, Tramp can let you do that, and it can even let you do that on remote machines where you aren't installing GNU Emacs to start with. This can help bypass some of the reasons that vi became my sysadmin's editor, assuming that you trust (your) GNU Emacs with holding and using sensitive passwords and probably have an instance running all the time. I don't use GNU Emacs that way (and I don't really trust it with sensitive passwords), so this aspect of Tramp isn't interesting to me. After years of using vi as a sysadmin's editor, I've become fluid in vim and I'd often rather use it than GNU Emacs.

However, the host of Wandering Thoughts doesn't have my Emacs or my Emacs environment, and I'm not going to put either on it. Not maintaining your Emacs environment on a remote host but still using Emacs to edit files there is a good use of Tramp, and for obvious reasons the whole experience is quite snappy when you're just editing text files; the only time a remote connection gets involved is when I save the file (which I actually do a lot, it's a reflex). I don't think Vim is better or worse than Emacs for editing entries, but they are different, and right now I apparently mildly prefer the modern GNU Emacs editing experience (in X, where I get nice red squiggle underlines for words the spell checker doesn't recognize and various other niceties).

(It's petty of me, but one irritation of local editing of entries with Vim is that I reflow paragraphs in Vim with '!}fmt', and the version of fmt on the host believes very firmly in two spaces after periods and I don't. I was forever going back to turn them into single spaces. GNU Emacs does it right for my tastes, although its reflow behavior is not quite what I want in other ways. There's definitely some aspects of the Vim editing experience I like better than the Emacs one.)

I've also been using the various editing actions I want for writing Wandering Thoughts entries as excuses to learn various GNU Emacs features, currently Rectangles. Rectangles are roughly the GNU Emacs equivalent of Vim's block visual selection mode and it's handy to know them, even if I find them somewhat more awkward than in Vim.

(There are so many keystrokes and every so often I will fumble a C-x r <something> operation and have to back to re-establish the rectangle selection I just lost. In the GNU Emacs way, there may be a package, command, or setting that would do this for me, but I'd have to find it.)

Suppose, hypothetically, that you have a service that works on stuff in a mounted filesystem and will malfunction if that filesystem goes away (for example, a media server that's working from a NAS-mounted drive). You'd like to immediately stop the service if the (remote) filesystem goes away for any reason so the service won't freak out (or has minimal chances of doing so). It's possible to do this in systemd but it's not quite obvious what you need to do and what the drawbacks are.

The simple and obvious approach is to set 'RequiresMountsFor=' for the directory you need. As the documentation says, this adds a Requires= and After= to the relevant systemd.mount unit for the filesystem. Unfortunately, this appealing basic solution has a number of limitations. The biggest one is that it only works at all if someone does an explicit 'systemctl stop <fs>.mount', not if the mount disappears through other means (for example, 'umount'). And it only works at all if the mount is explicitly listed (in /etc/fstab or as a .mount file), not if the mount is created on the fly by some other service (for example, if it's a ZFS mount). As far as I can tell, you can't fix this by setting After= to the service unit that winds up mounting the filesystem; you're just stuck.

(If you set ConditionPathExists= or one of its friends to something appropriate, your service won't start when the filesystem is missing. If it's mounted by a service, you'll definitely want to set an After=.)

If you directly set Requires= and After= to the .mount unit and the filesystem is mounted on the fly by some service, your own service won't start automatically (even after the mount appears, because systemd doesn't have triggering), but at least now 'systemctl stop <fs>.mount' will correctly stop your service. This is perfectly sensible on systemd's part; you told it that your service requires something that systemd doesn't know about and thus isn't there, so it can't start. To get your service to start on boot, you need an /etc/fstab entry or a .mount file.

Even with an /etc/fstab entry or .mount file, your service still won't stop if the filesystem is unmounted outside systemd. Systemd will correctly show the status of the <fs>.mount as 'inactive (dead)', but it won't stop your service despite the 'Requires='. It will stop your service if you then do an explicit 'systemctl stop <fs>.mount', even though this doesn't visibly change the status of the <fs>.mount unit.

To actually get your service to stop when the filesystem is unmounted outside of systemd, you need to set BindsTo= instead of Requires=. As the systemd documentation says, this is a very strong dependency (and also you want to have an /etc/fstab entry or a .mount file for the filesystem), but finally we have something that works.

If you have a BindsTo= to a .mount unit for a filesystem that's mounted by some other service and so doesn't have an /etc/fstab entry or a .mount file, your service won't start automatically (you have to arrange to start it after the filesystem and its .mount unit exist) but your service will shut down even if something does an 'umount <fs>'.

(One comment.)

Grafana is a metrics dashboard and visualization system, mostly commonly used as the display side of a full metrics and monitoring system. A while back on the Fediverse I praised it for good reason:

I may snark on Grafana the company, but Grafana the dashboard system is a genuine marvel. I have been around as a Unix sysadmin long enough that I've seen 90s and 00s era 'enterprise' monitoring and dashboard systems (from a distance). They cost a lot, were entirely closed and proprietary, and which didn't look half as good (or have half the functionality) as a Grafana dashboard I can build myself.

But Grafana is not merely an impressive piece of software; it's an impressive piece of web based software. Grafana dashboards are web pages and are typically built in a GUI through the web. So today I also said:

To extend this, Grafana the dashboard system is not only a genuine marvel, it's a genuine marvel of the (relatively) modern web, yes JavaScript and all. It has a ton of fluid interactions (both in viewing and building dashboards) that look quite a lot like native app features, but it's all done through the browser instead.

(And in the large scale of things it's not so long ago that our 'dashboard' systems were displaying hard-coded graphs by rendering PNGs on the server.)

It wasn't that long ago that the state of the art for web "dashboards" was hard-coded RRDtool or Xymon web pages that might display one or a few graphs that were created by rendering data on the server to PNGs and then displaying the PNGs in only slightly interactive HTML. If you wanted to change the time range, the server re-rendered new PNGs, and you usually picked the time range through some sort of form interface. This was a painful enough exercise that our old Xymon based monitoring system had almost no dashboards and almost no one ever looked at the dashboards we had.

When we moved from Xymon to Grafana (and Prometheus), the shift felt quite dramatic for me. It was sometimes hard to believe I could do what I was doing in a web application; it felt like magic (good magic). And my use of Grafana has been relatively basic. The university's central IT people built a Grafana dashboard for university wireless information which shows a map of each campus, overlaid with current counts of active wireless connections in each building, and you can select a building to see statistics over time for that building.

(The dashboards aren't available to unauthenticated people but the landing page is here, which means that the central IT people consider this good enough for public visibility. There's currently also an older status page where the usage graphs are the old 'rendered PNGs' experience.)

Of course people have been doing sophisticated web applications for a while (Google Maps is probably the starting point, or at least the most well known of them). But Grafana is open source software (although the company is VC funded) and it runs as a standalone program on ordinary hardware. And thanks to browsers enabling minority platforms I can use it from a Linux machine.

The modern web is sometimes a decidedly unpleasant place, with mandatory JavaScript shoving on-page popups and other things in your face, and intrusive advertising, and all of those ills. But it doesn't have to be that way and these (semi-)modern browser technologies can be used for good, to enable things that would have been at least infeasible in the old world.

(Since Grafana has been pretty capable for more than a decade now, it's hard to say it requires truly modern web technology. But I still think of this level of interactive JavaScript and drawing and so on, creating a native application like GUI interface, to be 'modern web' as opposed to the old HTML web that you find here on Wandering Thoughts.)

We have a number of Linux systems where we set strict memory overcommit handling and sometimes this can cause problems. Some years ago I speculated that we might want to fix the fundamental problem another way, and then earlier this year I investigated that a bit and determined it would likely work for us. This summer we rebuilt all of the relevant machines using Ubuntu 26.04 and you probably will not be surprised to know that we're still using strict memory overcommit (cf).

That's about a four year gap between having a clever idea and actually looking into the clever idea (and then still not acting on it). Part of that is inertia (strict memory overcommit mostly doesn't cause us problems, so we usually don't think about it), but a lot of that was losing track of my clever idea. This is undoubtedly not the first potentially useful idea I've had and lost track of, and it's probably not going to be the last.

Writing down a clever idea is necessary but not sufficient to keep track of it and eventually get around to further considering it and maybe investigating it (after all, I wrote my idea down here in an entry). What I didn't do is follow through on anything, and even when I did follow through it was only to take the idea one step further. It feels like I should do better than that, that good ideas shouldn't linger around for almost half a decade until they resurface almost coincidentally.

On the other hand, maybe this is an illusion. Even if a clever idea works (which isn't guaranteed), it may not solve a problem you (I) currently have. One reason I left this idea sit for so long is undoubtedly that we don't really have problems with our current strict memory overcommit settings, so there was nothing to push me to go look into potentially better solutions. There are always things I could be looking into but I have both limited time and limited motivation.

(This is one aspect of 'a change needs to be better, not just another way of getting the status quo'. And it has to be enough better to justify the investment of time and attention.)

I don't have an answer, but I found it interesting to actually observe myself having a clever idea and then not so much dropping it as losing track of it and never getting around to doing anything more. It feels like something I should pay at least some attention to.

(I could make a resolution to write down my clever ideas somewhere. But in practice, places I write down ideas rapidly become idea graveyards, because I never look at them again. I've learned the painful way that I have a limited 'top of mind' capacity and once things drop out of that, they disappear. This happens to my inbox, it happens to clever ideas files, it happens to a lot of stuff. Although writing ideas down means that I do have some chance of finding them again, instead of no chance, and seeing my old notes and thoughts.)

(2 comments.)

Not that long ago I wrote an entry on my views on Flymake and Flycheck in GNU Emacs, when Flycheck v36 was the latest release and my impression was that there hadn't been deep changes in Flycheck for a while. To summarize my views, the two were basically equal but Flymake was a bit better if you only used Eglot while Flycheck gave you more flexibility, especially if you weren't using a LSP client. All of that changed in Flycheck 38 (and Flycheck 39), which added a bunch of exciting and important features (cf). Now, my view is that you should pick Flycheck if you're at all on the fence.

Flymake is (still) perfectly okay when you're using a LSP client and for a few languages even without one, and it has the advantage of being built in to GNU Emacs (which means you don't need to obtain and configure it in order to get things to basically work, at least in Eglot). But Flycheck is now more or less better at everything. It has good built in Eglot integration (including features Flymake currently lacks), the new annotation mode can surface diagnostics just as well as Flymake, and Flycheck has pulled ahead on useful general features, including LSP server based diagnostics for people who don't want to use a LSP client. There are only a few small things left where Flymake (in Eglot) is ahead of Flycheck in features, and Flycheck still has its better support for multiple checkers and switching between them.

(As a quick tip to myself, if I want to switch back and forth between LSP provided diagnostics and native Flycheck checkers in an Eglot buffer, the easy thing to do is to turn flycheck-eglot-mode on or off. This is most useful in Python, where for me the native checkers are more linters and the LSP diagnostics are real error checkers.)

This isn't quite the situation with Eglot versus lsp-mode, where I think you'll wind up using Eglot sooner or later because it's better integrated into the overall GNU Emacs ecosystem. Instead I think Flycheck has simply become better, and it's likely to stay that way because it has the advantage of being able to make new releases faster than Flymake can, because Flymake releases are part of GNU Emacs releases and those only happen every so often. Sooner or later you're probably going to want something that Flycheck has and Flymake doesn't.

(This includes in Emacs Lisp, where I don't think Flymake has an equivalent of Flycheck's flycheck-emacs-lisp-initialize-packages.)

This big Flycheck improvement has caused me to drop a number of my GNU Emacs packages. Gone are flycheck-eglot (now a native feature), flycheck-inline (also now a native feature), flyover, and sideline along with all of its associated packages. In theory the latter two aren't completely obsolete; flyover works in Flymake as well, and sideline can expose LSP code actions. But in practice I wasn't using them and I decided to do some summer configuration cleaning.

(I haven't yet gone so far as to drop all of my Flymake configuration, partly because I want to be able to see what a native Flymake experience is like for comparison and debugging purposes.)

PS: Flycheck staying ahead of Flymake in the future isn't guaranteed. Flymake could have a renaissance that improves it rapidly (the recent Flycheck improvements might encourage that), and Flycheck could go quiet again. You can't predict the long term future of an open source project, especially in the world of GNU Emacs packages (and GNU Emacs itself).

We've recently wound up with some unusually big (x86) servers, from a vendor we don't normally get servers from. One of the unusual things about these servers is that they take a very long time to initialize on boot (at least by our standards). I haven't actually timed it, but I'm pretty certain there's more than a minute of staring at 'chipset initializing', and it may be as much as several minutes.

In a way this isn't entirely surprising, because modern systems are really complex under the hood (cf). Your RAM modules may require 'training' (also) to reach reasonable speeds (and these servers have 24 of them each), PCIe busses and devices require initialization and probing (and these servers have a lot of PCIe devices), and so on and so forth. It's not as if we haven't had relatively slow booting x86 servers before, and how much the BIOS tells you during this process is probably partly a design choice (although presumably you have to get a fair distance into the overall hardware initialization process before you can display much video).

(In theory I believe the firmware saves data from 'training' RAM so it doesn't necessarily have to go through the full process every time. Possibly there are circumstances where this doesn't work out in practice, or there are other things going on with these servers.)

In some ways a long firmware boot is just an inconvenience. It means rebooting or powering on one of these servers takes an extra long time, but that's not a big issue for these servers. We're not using them in a situation where how fast they take to reboot is all that important, although it is kind of annoying to start a reboot of a server then have to make a mental note to come back in a few minutes when it might have finished the process (it's easy to get distracted).

Where it gets painful is when you (I) need to interact with the BIOS firmware (or the Linux bootloader) in the brief window between when the firmware initializes enough to start accepting input, when it switches to the bootloader (GRUB for us), and when the bootloader brings up the default kernel, at which point it's too late to affect either. If I need to get into BIOS firmware setup, I get to sit watching the 'chipset initializing' for as long as it takes, and that's tedious (and painful if I miss my window of opportunity).

(I was going to say it would be nice if the server's BMC had an option to go into BIOS setup on a triggered reboot or power-on, but now that I look the BMC has a 'remote BIOS setup' feature. Amusingly, this works through a completely different web environment than the BMC's web environment. For example, the 'remote BIOS setup' sub-site uses HTTP Basic Authentication instead of the BMC's conventional cookie-based login process.)

PS: I'm pretty sure this isn't the slowest-starting firmware I've seen. Long ago I dealt with SGI servers that my memory asserts took at least five minutes to get through their firmware (cf), but they had the excuse that they had much, much slower MIPS CPUs.

(4 comments.)

Over on the Fediverse I said something:

It has been '0' days since I've been pointedly reminded¹ that URLs aren't strings.

¹ it wasn't painfully as such because the only thing tripping over it was Bingbot doing bad things, but.

Suppose that you have a URL in the form of a UTF-8 or Unicode text string and you want to show it to people in a web context, and some parts of the URL have characters (or Unicode codepoints) that fall outside of the straightforward 'US-ASCII' character range. If you're putting the URL (string) into HTML, you can often get away with not doing anything special (beyond using UTF-8); your HTML is probably UTF-8 and browsers will do all of the hard work for you. But if you're putting your URL in other places, you may have to encode and quote it. One of those places you need to encode it is in HTTP Location response headers, which you may generate if you're redirecting a URL to some canonical version of it (for example, one without random 'utm' query parameters).

(There are a few quoting issues in URLs in HTML links and so on that browsers can't sort out for you, but generally you can shovel in any old un-quoted, un-escaped thing and it will probably work out.)

Although browsers may not enforce it, HTTP headers are effectively restricted to US-ASCII or at least ISO-8859-1, which means that you should encode any Unicode or UTF-8 in URLs that appear in Location: (and your web framework may require you to do this). However, Unicode in different parts of the URL must be encoded differently. Unicode in the host portion is encoded using IDNA, while UTF-8 in the path or query parameters is %-encoded. And of course some special characters in the path (especially '?') also need to be %-encoded, while the '?' that separates the path from the query parameters can't be encoded at all (and this is also true for URLs being inserted into HTML).

If you treat an entire URL with scheme, host, path, and query parameters (and possibly a fragment identifier) as a string and try to encode it blindly, you will wind up with suffering. That's what happened to me when I tried to fix my problem the simple way and it blew up in my face, leading to my Fediverse post (more or less).

My specific problem was that Bingbot was occasionally making HTTP requests for URLs with random Unicode characters added to the path, directing that request to alternate, non-canonical hostnames for Wandering Thoughts. If Bingbot had asked for these URLs on the canonical hostname, it would have received a 404 response (no URL paths here have UTF-8) and everything would have been fine. But because Bingbot was asking for an alternate name, DWiki started out by doing a HTTP redirection to the canonical hostname and that redirection echoed Bingbot's mangled, UTF-8 encrusted URL back into the Location: header in the response. This caused a (Python 3) string encoding error when the low level response handler couldn't encode the headers to iso-8859-1.

Sidebar: Unicode versus UTF-8 bytes

Suppose that you get a request with a URL path that ends in '%E2%80%99'. After your web framework decodes this, there are two things you can wind up with at the end of the path; you can get three bytes of UTF-8, 0xE2 0x80 0x99, or you can get the Unicode codepoint U+2019, the character ’. You'll get the former if your framework doesn't think of URL paths as having a character set and simply undoes %-escapes, and you'll get the latter if your framework assumes UTF-8 encoding and turns the UTF-8 bytes into Unicode. (And then depending on your environment, both versions may print to your terminal and appear in logs the same way.)

(This assumes your language has a distinction between 'Unicode strings' and 'UTF-8 strings', which isn't always the case. Python 3 more or less does, and urllib.parse.unquote() defaults to UTF-8 as the character encoding of the URL path when it turns raw bytes into Unicode strings.)

Conversely, it may matter if you start with a Unicode URL or a UTF-8 URL. If you have a Unicode URL, on the one hand you have the correct starting point for hostnames for IDNA (which is not an encoded representation of UTF-8), but on the other hand you have to decide what character encoding your URL path and query parameters will be in (you'll probably pick UTF-8). If you have a UTF-8 URL, you can decide to directly %-encode the path and so on from your existing bytes, but you need to do something more involved for non-ASCII hostnames to produce the correct IDNA result.

(3 comments.)

I wrote the other day about an irritatingly incompatible change in Python 3.14's argparse, where if you deferred setting a default value until later, certain argparse things would now error out. In a comment, Ian Z aka nobrowser asked (in part):

Setting the default right in the add_argument call has always seemed more natural to me. Do you have a case where it is not possible or just inconvenient to do this?

Some of my usage of set_defaults is historical and I might do it differently today; in today's Python, setting default values in the add_argument call is clearly the accepted and expected style. However, I feel that using set_defaults for some or all settings can be a clearer choice if a bunch of default values are clearly related to each other. By setting related default values together, you give future readers visibility into what all of the defaults are, in one place, without forcing them to read the code and pick out all of the 'default=' arguments in add_argument calls.

In my view this is especially useful for quick views to remind yourself of what the code's default behavior is. Yes, you can run '--help' (if it prints default values, perhaps with ArgumentDefaultsHelpFormatter), but that tells you a lot of additional things beyond the defaults. How much this matters depends on how many arguments you have and how they manipulate their destinations (which isn't always simple). Some of my programs are definitely ones where the default values belong with argument, in add_argument, but others have enough that I feel that set_defaults is perfectly fine and sometimes easier to deal with.

Using set_defaults also puts the name of the option and its default value next to each other, as an assignment. In an add_argument call they're split across two keywords:

   p.add_argument("--whatever", dest="whatever",
                  action="store_true",
                  default=False,
                  help="Don't do whatever.")

  # Compare to:
  p.set_defaults(whatever=False, ...)

To read the add_argument() you (I) have to combine 'dest="whatever"' with 'default=False' to get to 'whatever is False by default'. By contrast, that's right there in the set_defaults.

(Looking back and forth between various programs actually makes me feel I haven't been using set_defaults as much as I probably should. Putting the default value into each new add_argument call is the easy way as my programs evolve, but maybe I want to go back more often and set all of them in one block once enough arguments have accumulated.)

Sidebar: A slightly tricky destination manipulation

Consider the following code (which is a version of something I've done in some of my programs):

   p.add_argument("--no-whatever", dest="whatever",
                  action="store_false",
                  default=True,
                  help="Don't do whatever.")

This is a perfectly reasonable way to turn off something but it's a little bit confusing to read 'default=True' in the context of setting up the '--no-whatever' argument. In my view this sort of thing is clearer in set_defaults, where you aren't distracted by other things and it's clearer that the default value of 'whatever' is True (on).

(This trick also causes ArgumentDefaultsHelpFormatter to report the misleading '(default: True)' for '--no-whatever'.)

What's now a long time ago I wrote an entry about an oddity in our network setup that was the result of historical evolution. At the time we had our printers on a physically separate set of switches, which carried a single active VLAN for a private sandbox network that all of the printers were on. These switches had shrunk from their original purpose of carrying several of our public networks that weren't on our normal core switch environments because they'd started out as a "protocol based VLAN" on our very old switches, a feature our new(er) core switches didn't support, and we thought we'd had decent reasons to not split the public networks into real separate VLANs.

Well, time marches on and we do turn over network gear and replace old switches, so you can probably guess what has happened since that entry. These days our 'printers' private sandbox network is merely another VLAN carried on our regular switch fabric, with all of its old switch infrastructure long since retired. We also bowed to the forces of convenience and put all of our public networks to our core switches as tagged VLANs, not just one of them. In the process I believe we changed the VLAN number for the primary network of our old "protocol based VLAN" setup, which probably happened as we turned over core switches. Exactly when all of this happened isn't clear to me at this late date, but it may have been starting even at the time when I wrote the original entry.

(That all of this is fuzzy to me at this point is another example of how I'm not going to accurately remember our past setups and thus we should periodically write either or both of end of service notes or periodic overviews and descriptions of our current (network) environment.)

Large scale changes in network setups can be hard to do all at once (or at least require some degree of downtime), but sometimes you can do them incrementally as you deploy and replace switches. Plus, there are hacks to do things like run old and new VLAN numbers in parallel, with one set on old infrastructure and a new set on new infrastructure, crossing over from the old to the new infrastructure through things like untagged ports (if you have only a few VLANs that are renumbering).

It's also worth remembering (even if only for myself for future use) that today's awkward, historical network and other situations aren't something you're stuck with forever. Even if you don't want to make a 'big bang' style disruptive change, you can probably fix the situation slowly over (enough) time. The process of changing over may be awkward (as our printers network was, as covered in the original entry), but things can evolve.

(This does require that you have a long enough planning horizon to carry this all out. We're lucky enough that we can do this sort of thing; other environments aren't as stable as ours. But our infrastructure is surprisingly stable.)

Today I discovered that Python 3.14 contains an irritating, incompatible change to historical argparse behavior, which I can best illustrate with a little piece of code.

import argparse
def demo():
    p = argparse.ArgumentParser()
    p.add_argument("-n", dest="anumber", type=int,
                   action='store', metavar="NUM",
                   help="A number, default %(default)d")
    p.set_defaults(anumber = 1)
    o = p.parse_args()
    print("Parsed.")
if __name__ == "__main__":
    demo()

Before Python 3.14, this program would run successfully (including with '--help', which will report that the default number is '1'). In Python 3.14 and later, this program will fail with a Python exception from argparse in _check_help() that runs more or less:

[...]
TypeError: %d format: a real number is required, not NoneType
[...]
ValueError: badly formed help string

This change is not in the 3.14 release notes for argparse, but with digging you can find that it's from gh-124899 aka gh-65865, which is described as "Raise early errors for invalid help strings in argparse". This is a perfectly good change with good intentions, but it has a problem.

The problem is that through Python 3.13, it was perfectly valid to refer to things like '%(default)' in your help text but not have the default value set until later, in a following .set_defaults() call that set the default value of a block of related options (or all of them). After gh-124899, this will fail if you use any formatting option for '%(default)' that can't be satisfied with None, because the help text is formatted immediately and so has the unspecified default value of None. This doesn't happen if you use '%(default)s' for everything, because a None can be formatted as a string.

Currently this restriction isn't documented for help=, but all of the examples use string formatting, ie '%(default)s', even for something that's an int, and also the example sets a default= in the add_argument() call.

Unfortunately I don't really see an easy and clean way to change this situation. If argparse doesn't check the help text immediately, there's no guarantee that it will have a chance to do so later, before you parse arguments (you might not even call .set_defaults()). In theory argparse could make up a temporary default value when checking the help string (if you use '%(default)' in it), but in practice this is at least somewhat complex and might hide errors if the default is never set. In a way the cleanest fix would be to make all uses of '%(default)' with an unset default be an error (and document this), but that would be a clear API break.

(Since Python 3.14 has shipped with this issue and it's survived through 3.14.7 as far as I know, the overall change to check help strings early probably isn't going to be reverted any time soon. I'm probably a highly unusual person in combining .set_defaults() with using '%(default)d' instead of formatting all default values as strings.)

(3 comments.)

As a system administrator, I spend a certain amount of time spelunking through the history of some particular file in Git because I'm trying to understand how it evolved, what it used to do, when something was introduced or removed, and so on. There are two ways to do this, either through a straightforward history of a file or through what I'll call the blame history of a file, where you start with 'git blame' and then ask for what the file looked like before a particular change of interest. I'm pretty sure I first saw the blame history view on Github, where it's a signature feature of their blame view (in the form of the 'Blame prior to change ...' option).

In GNU Emacs you can look through a file's straight history with git-timemachine, or by using 'n' and 'p' in VC's annotated view of the file (C-x v g); if you're using the annotated view purely to page through versions of the file, you may want to use 'v' to turn off the commit information on the left side. In Magit, looking through versions of a file over time is done by starting with magit-blob-previous (often 'C-c M-g p', cf), and then 'p' and 'n' (and 'q').

Both VC and Magit can provide a blame history view of a file, and which one is easier to use will depend on your views. In VC, this is done through vc-annotate, C-x v g normally, and in Magit it is done through (of course) magit-blame (which is also accessible through git-timemachine and Magit's blob view of a past version of the file, which will let you start the blame at a particular version), or you may want to jump directly to magit-blame-addition. In Magit's blaming mode you jump to the file before a particular change with 'b' and return with 'q', and you can cycle through the blame display styles with 'c'.

In VC's annotate mode you jump to the (annotated) file before a particular change with 'a' and you can turn the 'git blame' style leading information on and off with 'v'. You can also jump to the file as of a particular change with 'j', which is potentially handy and which I don't think Magit supports directly (although you can copy the commit hash with M-w and then use magit-find-file or friends). However, as far as I know VC's annotate mode doesn't have an easy way to return to the blame view you were at before you used 'a' or 'j'. Moving through blame history in VC's annotate mode appears to be mostly a one way trip.

Neither VC nor Magit quite provides the Github like experience. Magit lets you navigate back, the way Github does, but it doesn't have a 'git blame' style display of the commit information on the left side. VC has the 'git blame' style display but not navigating back. For my purposes I probably care more about navigating back in the case of mistakes than blame style display, so I think I'll turn first to Magit.

(git blame can more or less do this from the command line, since you can start it for a file from a specific version. This gives you both the VC 'j' and 'a' commands; for 'j' you use the commit that 'git blame' reports for a line, and for 'a' you use '<commit>~' for one before that commit.)

(This is the kind of entry I write because it pushes me to do a certain amount of research and then write down what I've learned for future use.)

Many years ago I wrote an entry about a deep dive into the Linux OS memory use of a simple Go 1.11 program. For reasons of my own, I'm returning to this in Go 1.27rc2, because these days Go can tell us much more information about memory usage. As before, I'll use this very simple program:

package main

func main() {
    var i uint64
    for {
        i++
    }
}

You'll want to set 'go 1.25' or higher in the go.mod file, and then when you run this on a suitable Linux distribution you'll get a quite informative /proc/<pid>/maps file, which I'm going to break up into sections and reformat a bit to have sizes instead of start-end runs. All of the following is from a 64-bit x86 Ubuntu 26.04 LTS server.

First we have the program's machine code, read-only data, initialized variables, and zero'd data ('bss') space:

00400000          504K r-xp   memdemo
0047e000          656K r--p   memdemo
00522000           32K rw-p   memdemo
0052a000          212K rw-p 

This is all contiguous, covering 0x00400000 to 0x0055f000. The 'r', 'w', and 'x' flags mean read, write, and code execution respectively, and the 'p' flag means that all of these mappings are copy on write ('private'). I don't know why the code and read-only data sections are mapped 'p', and I'm not going to speculate.

(If we compare to the Go 1.11 version from 2018, it looks like the program itself has grown, probably through growth in various runtime packages that are implicitly pulled in.)

Then we have the main Go heap with its initial 64 MByte arena. This program has apparently only needed a 4 MByte chunk out of it to actually be allocated, with the rest as reserved space right now.

d75e4000000     12288K ---p  [anon: Go: heap reservation]
d75e4c00000      4096K rw-p  [anon: Go: heap]
d75e5000000     49152K ---p  [anon: Go: heap reservation]

This is all contiguous in a block from 0xd75e4000000 through 0xd75e8000000, which makes the 64 MByte nature more obvious. The Go heap's location used to be fixed, but since Go 1.26 randomizing the heap base address has been the default on 64-bit platforms. You can in theory turn it off for now because it's a lingering Go experiment (via).

(This is RandomizedHeapBase64 in internal/goexperiment.)

Then we have an assortment of Go internal allocations, all stacked up one after the other:

7b3b11460000      256K rw-p  [anon: Go: immortal metadata]
7b3b114a0000     1408K rw-p  [anon: Go: profiler hash buckets]
7b3b11600000    32768K rw-p  [anon: Go: heap index]
7b3b13600000   289708K ---p  [anon: Go: scavenge index]
7b3b250eb000        4K rw-p  [anon: Go: scavenge index]
7b3b250ec000   234576K ---p  [anon: Go: scavenge index]
7b3b33600000   289708K ---p  [anon: Go: page summary]
7b3b450eb000        4K rw-p  [anon: Go: page alloc]
7b3b450ec000   270788K ---p  [anon: Go: page summary]
7b3b5595d000        4K rw-p  [anon: Go: page alloc]
7b3b5595e000    33844K ---p  [anon: Go: page summary]
7b3b57a6b000        4K rw-p  [anon: Go: page alloc]
7b3b57a6c000     3664K ---p  [anon: Go: page summary]
7b3b57e1c000      512K rw-p  [anon: Go: immortal metadata]
7b3b57e9c000       64K rw-p  [anon: Go: gc bits]
7b3b57eac000       64K rw-p  [anon: Go: allspans array]
7b3b57ebc000     1024K rw-p  [anon: Go: page alloc index]
7b3b57fbc000       72K rw-p  [anon: Go: immortal metadata]
7b3b57fce000      564K ---p  [anon: Go: page summary]
7b3b5805b000        4K rw-p  [anon: Go: page alloc]
7b3b5805c000      456K ---p  [anon: Go: page summary]
7b3b580ce000      128K rw-p  [anon: Go: page alloc]
7b3b580ee000      256K rw-p  [anon: Go: immortal metadata]

This area runs from 0x7b3b11460000 through 0x7b3b5812e000, although the largest bits of it aren't accessible ('---p', they have no read, write, or code execution permissions). This is a bit over a gigabyte of address space (1159992K, specifically). As you might suspect, the 'page alloc' pages are carved out of an overall 'page summary' block of memory, and this and related things are found in runtime/mpagealloc.go and runtime/mpagealloc_64bit.go. The 'scavenge index' is from runtime/mgcscavenge.go, although the name is assigned to the mapping in runtime/mpagealloc_64bit.go (page allocation and scavenging are connected to each other). The 'heap index' is from runtime/malloc.go; everything in this category (of which there are probably multiple allocations merged together) is a 'L2 arena map'.

This is a lot more reserved address space than Go 1.11 used back in 2018, but we can also see that only about 36572 KBytes of it is actually allocated as accessible memory (with read and write permissions), which is not much more than Go 1.11 used. This difference between simply fencing off some address space and actually allocating memory into it can matter a lot in some circumstances.

(Not shown is how much of that memory has actually been touched and is either resident or dirty. Programs often establish read/write memory mappings without touching every page of them.)

Next is the special kernel 'vdso' (also) support, which the kernel maps into everyone's address space, in a block from 0x7b3b5812e000 to 0x7b3b58136000:

7b3b5812e000       16K r--p  [vvar]
7b3b58132000        8K r--p  [vvar_vclock]
7b3b58134000        8K r-xp  [vdso]

Then the official process stack, which is initially allocated by the kernel when it exec()s the process, before Go is involved:

7ffc60b8f000      136K rw-p  [stack]

And then we have a special 'vsyscall' area (which is apparently a legacy thing):

ffffffffff600000    4K --xp  [vsyscall]

On suitable Linux systems, Go sets the names of all of these areas using a function in runtime/set_vma_name_linux.go under the right circumstances. In general, this is controlled by the decoratemappings option for $GODEBUG, which was added in Go 1.25 and defaults to being on in that version and later. However, simply building your program with Go 1.25 or later isn't sufficient, as I found out; if your go.mod names an earlier Go version, Go will default to decoratemappings=0 even if you build your Go program with a later version.

In addition to building your program with a modern Go version and either having Go 1.25 or later set in go.mod or explicitly setting GODEBUG=decoratemappings=1 when you run your program, you need to be using a Linux kernel with this feature enabled. This is controlled by the kernel configuration option CONFIG_ANON_VMA_NAME and different distributions set it differently. Ubuntu has turned it on since at least 22.04 LTS (the oldest version I have access to right now), while Fedora has it off.

(Apparently the reason to leave it off is that it prevents merging adjacent virtual memory areas with the same permissions but that have different names. For example, the first two memory areas at 0x7b3b11460000 seem to get collapsed into one with 'decoratemappings=0'.)

Suppose, not hypothetically, that you're operating a compute server and its kernel is logging a series of messages to the effect of:

__vm_enough_memory: pid: 719830, comm: node, bytes: 221184 not enough memory for the allocation

(The PID, the 'comm:' command name, and the number of bytes will vary.)

This particular function has cropped up before, and this message from it means that you're hitting a limit on how much memory the kernel will let the process ask for under the current circumstances. Generally this means you're operating in strict overcommit mode, because it's hard (but not impossible) to get the Linux kernel to refuse to give you memory otherwise.

(In the default heuristic overcommit mode, a process has to ask the kernel for more memory in one request than it has total RAM plus swap before the kernel will turn you down. This is a pretty extreme and unlikely situation unless you have a tiny system.)

As the kernel source comment on __vm_enough_memory() says (currently here in mm/util.c), this check is triggered when a process attempts to allocate a new 'virtual mapping' (which includes when it wants to grow one). In modern programs, this will most commonly happen through mmap() and most commonly when the program wants to allocate more dynamic memory, but this can also happen on (attempted) stack growth or growing the 'break', which is still done by some programs in some circumstances.

Currently, there are two ways to be out of memory in __vm_enough_memory(). The first is to have no committed address space left at all, minus admin_reserve_kbytes if you're not 'root' (more or less). The second is to be a single process and to be trying to grow into the last bit of the commit limit. The details are more or less covered in the documentation for user_reserve_kbytes; to rephrase the documentation, if you have a large process that's trying to grow, typically this will mean that the process effectively sees an overcommit limit that is 128 MBytes below the true commit limit.

(A 'large' process here is one that has 4 GBytes or more of committed address space. These days that's not necessarily a genuinely large process.)

In either case, to trigger this message in strict overcommit mode your system usually needs to already be quite close to its overall limit on committed address space. It's entirely possible for this to be a quite transient thing, if one process is allocating a burst of memory and driving up the committed address space, then stops when it hits the limit and releases a lot of memory back (perhaps because it exited due to memory allocation failure, or crashed). A short term situation like this might not show up in your metrics system.

How easily and often programs run into this error depends on how much memory they ask for at once, in a single request. The initial message was about a comparatively very small allocation (216 KBytes) and that not being available means that the system was very close to its overall commit limit at the time, even accounting for the effects of user_reserve_kbytes. By contrast, consider the following message as an extreme case:

__vm_enough_memory: pid: 870355, comm: <redacted>, bytes: 137438953472 not enough memory for the allocation

That's a request for a 128 GByte allocation. The particular server this kernel message is from only has 128 GBytes of RAM total and after overheads, has a commit limit of 122.5 GBytes, so this program's allocation is never going to succeed as long as strict overcommit is on (and this particular program appears to have a habit of asking for 128 GBytes of RAM, it shows up repeatedly in the logs with different PIDs). A program that can never succeed in its allocation is an extreme case, but obviously programs that ask for tens of gigabytes at once are more likely to reach the limit than programs that ask for, say, 64 MBytes at a time (cf).

(The kernel doesn't attempt to split up allocation requests and give you part of what you asked for. If you ask for a 32 GByte mmap(), you either get it all or you get nothing. If the system has strict overcommit and 20 GBytes of commit limit left, you get nothing, while the program that asked for 64 MBytes gets its 64 MBytes and can keep asking for another 64 MBytes again and again until it gets almost all of those 20 GBytes.)

(One comment.)

Sometimes you have to try to understand what's going on inside a shell script or a collection of them. If you're lucky, you may be able to easily get a 'sh -x' output from the script or even scripts. This can be very enlightening but at the same time it can hide some things or at least make them much harder to see in all of the clutter.

The first and biggest thing that 'sh -x' doesn't give you is shell redirections. If the script makes heavy use of redirecting things into working files that it manipulates, shoves around, and then deletes, a lot of the specifics of this may be invisible to your debugging (although you can usually tell that redirection is going on). This is most likely to apply if the file names are set up dynamically through things like shell variables.

(Perhaps the simplest way of getting 'sh -x' to report shell variables is to use the ':' command, which does nothing but which does appear in 'sh -x' output. So you can use this as ': $VAR1 $VAR2 ...' to create something that only shows up with 'sh -x' and that fits in naturally with the rest of the 'sh -x' output.)

The next thing is that while 'sh -x' reports all of the commands that are run in a shell pipeline, it doesn't particularly report that they're run in a pipeline. Sometimes this is obvious and sometimes it's not. If you're using Bash (including if /bin/sh is Bash), you can use $PS4 to add more information to 'sh -x' output that may make this clearer, for example including '$LINENO'. Including the line number may also make it easier for you (me) to trace the flow of execution of a third party script.

(At least in Bash, 'sh -x' does report commands run in '$(...)' and so on, and in a distinct way. But as with everything else, it doesn't show you the context of that subshell.)

Sometimes information is passed around between parts of the script (or the script and sub-scripts) through environment variables rather than command line arguments. In theory 'sh -x' makes it possible to know and follow these, because it does print every variable assignment (including the final value). In practice environment variables may be set some distance (in time and space) from when they're used, so reconstructing things may take work. I don't have a good solution to this other than modifying the script to use things like the ':' trick to report variable values closer to when they're used.

(In theory you can use 'sh -xv' to get around some of this. In practice I find the resulting output to be quite hard to read.)

(Mostly I'm writing this down so perhaps I can remember what 'sh -x' isn't showing me and isn't going to show me the next time I run into a similar situation.)

(3 comments.)

Recently an interesting commit landed in the GNU Emacs Flycheck package, which shows diagnostics obtained from 'checkers', including Emacs language server clients such as Eglot (which are in turn getting their diagnostics from language servers for various languages). In the commit (also), Flycheck was updated to better handle a diagnostic source that reported information asynchronously; one such source is Eglot, partly because the diagnostics come from talking with language server processes.

The bug report for this stems from something I reported, and then when the core bug I reported was fixed, I reported a side effect of the fix. When I made the initial bug report I reproduced the issue both with and without Eglot, because I wanted to be sure that this wasn't specific to Flycheck's integration with Eglot. Since the initial issue was independent of Eglot, when I saw that the fix had a side effect I didn't bother to redo the dual check and tested in Eglot because it was most convenient.

(I have a Go project with a known issue Flycheck will report and I normally use Eglot with Go code. It's a lingering issue because I disagree with the specific lint diagnostic.)

I assumed that because the initial issue hadn't depended on Eglot versus non-Eglot, the new problem didn't either. This turned out to be incorrect because of a detail I hadn't known before I saw it in the commit, which is that Eglot produces diagnostics asynchronously while Flycheck's conventional checkers produce them synchronously (from Flycheck's perspective, although the checking happens in the background from your main Emacs work). If I'd known this detail I'd have checked in both situations but since I didn't, I assumed that all sources of diagnostics were functionally the same and I could test in whichever was most convenient.

I knew that bug reports were hard and that details could matter, and I still made an assumption when making my report of the side effect, which was in effect a bug report about a new issue. The assumption seemed reasonable based on what I knew, but I also didn't know all the inner details of how the system operated and one of those inner details turned out to matter and make a difference.

You (I) can and should draw an analogy to people reporting problems in our systems. I know the inner details of how our systems work and what details (probably) matter and which ones don't, but the people who are using our systems don't know either. It should not be a surprise to me to get problem reports that have irrelevant information in them but omit important details. What's happening is that people are guessing or making assumptions about what details matter and which ones don't, just like I did. And just as I did, maybe the person was fully detailed in their initial problem report but then when there are follow-up problems they start making assumptions about what they can not bother checking, trying, and so on.

(Because let's face it, doing all of those checks and looking at everything and so on is work, and if it turns out to be pointless you'll feel annoyed about it. At a certain point people stop doing work that only ends up annoying them.)

Applications to the experiences we system administrators have when filing problem reports with various organizations are left as an exercise to the reader. I may feel a bit more sympathetic to the organizations now, since they're dealing with this too (but certain behaviors are still quite irritating).

On the Fediverse, someone recently ran a poll about whether people could live without their (smart)phone. I said that I could (assuming that magically there were no mandatory phone apps) but that it would be annoying, and one of the reasons was that my phone is the best alarm clock I've ever had (as I got rubbed in my nose recently, although it's not flawless). Afterwards, I started thinking about why my phone's alarm is so good or at least so much to my tastes.

The simple, obvious, and more or less wrong answer is because my phone is a computer. This is more or less wrong because these days almost everything is a computer, or at least has a general purpose computer inside it; your flashlight might be a computer (and perhaps runs an open source flashlight OS), for example. This has happened because using a tiny cheap computer is often the simplest, cheapest, and fastest way to implement features (rather than designing a bespoke electronic circuit). So a physical, dedicated alarm clock could certainly be a computer and some of them probably are.

My taste in alarm clocks is silence and darkness; noises or light make it hard for me to sleep. Many physical alarm clocks glow, tick, or both. My phone doesn't do this but that's somewhat an accident of history. Phones started out not being able to afford the power to keep their screens on and lit all the time, or regularly making noises, and so you could turn on an alarm, set the phone down, and it would naturally be dark and silent until the alarm went off. Some modern phones have enough power and clever enough displays that they can keep a display lit up (and they're very proud of it so they often do it by default), but you can disable this and get back to a silent dark object.

(I believe that electric alarm clocks have traditionally been plugged into the wall partly to use wall electricity for time keeping; the grid frequency is extremely stable. Since they are plugged in they could use that wall power for all sorts of glowing and noise-making.)

Another part of the phone being such a good alarm clock is that an alarm clock application gets to use a large, general purpose, highly capable interface in the form of the phone's touchscreen. This allows an alarm clock application to expose a lot of functionality in an easily discovered form. I have named alarms that repeat on various different days of the week at different times in the morning and I turn on and off as required, and all of this is straightforward to see, keep track of, and manage. A physical alarm clock computer could implement all of the same software features of repeating alarms, day of the week alarms with different times, and so on, but it would struggle to expose them to people in a way that people could easily interact with. If you only have a simple physical interface in the form of some buttons and a limited display (and maybe a dial or two), in practice you're limited to features that are simple to express through those physical interfaces. Very few people will love a physical clock so much that they'll spend five or ten minutes laboriously setting up a new alarm through what we could call a 'narrow' interface.

(The cheat for this is to delegate the complex interface to an app on your phone, which then talks to the device with more limited input and UI capabilities over, say, Bluetooth.)

The phone's general purpose touchscreen interface isn't necessarily ideal for any specific thing (I have a simple kitchen clock with only a few buttons that's much easier to use for what I want than the phone's timer stuff). But it does let you have a lot of quite capable interfaces for a whole assortment of different (virtual) devices, among other sorts of software.

(One comment.)

Suppose, not hypothetically, that you have some Ubuntu servers that have local data filesystems (with stuff you want to keep) on separate disks from your system disks. These servers might be running Ubuntu 22.04 LTS or 24.04 LTS, and you'd like to update them to 26.04 LTS. Since you now have a kexec based network reinstall system, reinstalling them in place is attractive, but you need to figure out how to not destroy your data filesystems during the reinstall.

The 26.04 server installer offers some attractive options for this. It will recognize your existing software RAID arrays so you don't have to touch them, and then you can tell the installer to mount the extra RAID array somewhere as an ext4 filesystem and not format it. Well, in theory you can. In practice, the installer seems to always reformat anything you've told it to mount somewhere, even if you tried to tell it not to do so. I don't know if this is a bug or if it's me holding things wrong, but either way I don't think it's safe to let the installer touch anything you want to stay intact.

(The installer can definitely leave disks totally alone, because otherwise it would destroy its own USB memory stick when it's running from one. But you normally don't touch the USB memory stick at all in the installer.)

If you really care about the data, for example if it's years of historical metrics data and you're reinstalling the server in place, I think you (we) want to physically remove the data disks before hand. But if you're going to remove the data disks and you have the extra hardware (which we do), you probably want to install on new hardware and then move the data disks over. Certainly I'd have to test an in-place reinstall first to make sure the disks and the software RAID array would stay untouched, and I'm not sure I'd fully trust it even then, because who knows what the installer might decide to do under some situation I don't realize.

Even if everything works out, you'll probably want to correct the software RAID array's name by adding it to /etc/mdadm/mdadm.conf so it doesn't get called 'md127' or whatever. And you'll want to preserve things like the old system's /etc/fstab so you know what UUID to use to mount the filesystem on the data disks (or filesystems, if there's more than one).

(One reason I'm considering this crazy idea in the first place is my virtualization host for testing stuff is running Ubuntu 22.04 and needs an upgrade. I know how to move its setup around, but it would be so much easier if I could just network reinstall it, mount /virt again to get the VM images, and put a few configuration and data files back into place. I could do the entire reinstall from my office and much faster than any other way.)

Python 2 is famously relaxed about mixing tabs and spaces in Python code, although at the same time I believe it rigidly assumes that tabs are always at 8-space intervals (some editors offer you options here). When people used 8-space indent levels this wasn't too big of an issue, but it became one as the Python style moved to 4-space indents, because then some indents could be pure tabs but others had to involve spaces. Python 3 famously stopped being so relaxed, but for years I vaguely misunderstood how and thought it was more or less required to indent Python 3 code with spaces only.

(This belief wouldn't have survived if I actually thought about it, because I've converted historical Python 2 code that used tabs for indentation to Python 3 code with minor syntax changes and a change in the '#!' line. This definitely wouldn't have worked if Python 3 didn't still accept tabs.)

The official rules are described in the language reference in 2.1.8 Indentation. The documentation phrases it as:

Indentation is rejected as inconsistent if a source file mixes tabs and spaces in a way that makes the meaning dependent on the worth of a tab in spaces; a TabError is raised in that case.

If I'm understanding it correctly this time around, this means that you can have spaces after tabs, but you can't sometimes have tabs at the start and sometimes have the equivalent amount of spaces (well, equivalent in an 8-space tab world). The effect of this is that if you have some lines indented with tabs at the start or some lines indented by spaces only, you must indent all lines that way.

Where this comes up for me is when I'm editing existing code and for one reason or another my editor isn't set right. In GNU Emacs, this typically means that I wound up with indent-tabs-mode set wrong (perhaps my code for automatically determining the right setting didn't run). In Vi(m), this typically means I'm editing a Python 3 program written with spaces based indentation and I used the TAB key when adding a new line (which inserts an actual tab character for me). Sadly this somewhat discourages me from using Vim to edit Python 3 code.

In theory I could fix this as a one time thing by converting from tabs to spaces any time I touch a file. Vim makes this very easy, as I can simply run the entire file through 'expand' (with the ':!' command), and I can go the other way with 'unexpand'. GNU Emacs similarly has 'untabify' and 'tabify' commands (which I'm noting down here partly so I can find them later).

In practice this isn't likely to happen. I'm somewhat stubborn, sometimes it would be a bunch of work due to how many files are involved, and I don't want to yank around code and create a big mystery diff in the code's version control history. The time to do such a major conversion might have been as part of the Python 2 to Python 3 change (although as a separate commit from the meaningful changes), perhaps at the same time as changing from 8-space indents to 4-space indents. But generally that ship has sailed now.

(Keeping code tab-based has the quiet advantage that co-workers using plain vi(m) are more likely to be able to make successful spot changes to the code. If I dutifully convert it all to space-based indentation, hitting TAB in vi(m) is a trap.)

(3 comments.)

Over on the Fediverse, I said something a bit weird (and possibly incoherent and wrong):

My GNU Emacs tiny kingdom for something that only ran at byte-compile time purely so I can pacify ELisp checker errors that only happen then (in a file that is not byte compiled otherwise), and which can't be guarded with 'eval-while-compile' because then they get run twice.

(It's complicated. This file is load-file'd by my .emacs and needs macros from third party packages to pass type checking.)

I will skip ahead to the solution; since I normally use Flycheck, I fixed this by setting flycheck-emacs-lisp-initialize-packages to t, which works in my particular setup and for what I do with Emacs Lisp (but might not for you). The rest of this entry is the background explanation.

A while back I split up my .emacs, and some of the split was moving related blocks of use-package stuff to a number of separate files and then pulling each file in with 'load-file'. In some of the use-package blocks I have functions related to the package in either :init or :config sections. For example:

(use-package marginalia
  :init
    (defun marginalia-annotate-variable-docstring (cand)
    "Annotate variable CAND with only its documentation string."
    (when-let* ((sym (intern-soft cand)))
      (marginalia--fields
       ((or (documentation-property sym 'variable-documentation)
            (marginalia--definition-prefix sym))
	:truncate 1.0 :face 'marginalia-documentation))))
  [...]

When you're writing GNU Emacs Lisp code it's very handy to use either Flycheck or Flymake (I prefer Flycheck) to get diagnostics, so you can spot problems in advance. This works by running the Emacs Lisp byte compiler on your ELisp file and reporting any warnings and so on that it emits.

In general, to get a startup file like this to check properly, you need use-package itself to be available during byte compilation. The standard magic way to do this is:

(eval-when-compile
  (require 'use-package))

However, there's a complication; marginalia--fields is a macro and the syntax it takes is different enough to trigger a byte compilation error if it's interpreted as a function instead. If the macro was defined at byte compilation time, there would be no problem, but even with use-package available, the macro isn't pulled in because of (the lack of) package initialization. Marginalia is a third party package that has its Lisp files in an ever-changing subdirectory in ~/.emacs.d/elpa/, and that's not on the Emacs load-path until package initialization happens, so although use-package is active it can't find and load Marginalia to get the macro defined.

So, the clever person thinks, much like we force use-package to be pulled in during byte compilation, we can also force package initialization:

(eval-when-compile
  (package-initialize))

If I put this in the ELisp file with the above code, now the code passes byte compilation checks. However, if I start Emacs regularly, I get a warning:

Warning (package): Unnecessary call to 'package-initialize' in init file

This is happening because I also have a package-initialize in my .emacs file (somewhat for historical reasons). When my Emacs starts, both .emacs and this file are loaded, both run their package-initialize, and I get the complaint. When I made my Fediverse post, I was imagining a version of 'eval-when-compile' that silently didn't do anything when interpreted, so the guarded package-initialize would only run during byte compile checks, not during Emacs startup.

(I'm already not byte compiling these files normally, but obviously this is a bit dangerous. Which is sort of a hint that this was the wrong approach.)

Flycheck's emacs-lisp checker can automatically initialize packages at the start of its byte compilation check, but it normally does this only for files in user-emacs-directory (normally ~/.emacs.d), and I don't put my startup files there for various reasons. Setting flycheck-emacs-lisp-initialize-packages to 't' makes Flycheck do it all the time. I don't know if forcing Flycheck to initialize packages all the time may have side effects if you wind up working on Emacs Lisp things that aren't part of your startup files. This isn't an issue for me since I don't write Emacs Lisp outside of that sort of stuff.

Sidebar: Fixing this the more clever way

Flycheck has a function to decide whether or not a file is under your user Emacs directory, flycheck-in-user-emacs-directory-p. The morally correct way to fix my issue would be to use advice-add to also have it return true for files in the relevant bit of my personal Emacs Lisp directory tree, and to leave flycheck-emacs-lisp-initialize-packages at its default 'auto' setting.

(I'm probably not energetic enough to write the code necessary. It's possible that flycheck-emacs-lisp-package-user-dir is what I want to change, but I'm not clear what the effects of that are and I'm wary of tampering with it.)

Suppose, not hypothetically, that you're doing something inside a systemd service; for example, it runs a script with some environment variables set, and you want to get a full dump of those environment variables. My traditional approach is to write these to /tmp, but this doesn't work if the service is using PrivateTmp=yes (cf). Well, doing this doesn't put the resulting files directly in the regular /tmp.

For services specifically using PrivateTmp=yes, systemd puts their /tmp in /tmp/systemd-private-<hex>-<service>-<jumble>/tmp. The large <hex> value seems to be constant across all services, while the <jumble> is random. A service's /var/tmp is handled similarly, with the systemd-private directory in /var/tmp. This is sort of documented in the systemd.exec manual page:

If "true", the backing storage of the private temporary directories will remain on the host's /tmp/ and /var/tmp/ directories. [...]

You can also set 'PrivateTmp=disconnected' to give the service a completely detached /tmp on a new tmpfs; this is also implied by DynamicUser=yes. If you need to look at the /tmp of such a program (including one that implicitly has this setting by using DynamicUser=yes), I think the easiest way is 'nsenter -t <pid> -m' (as root), which will start a shell with that 'mount' namespace so its /tmp is the program's /tmp. I don't believe this disconnected private /tmp is mounted or otherwise available outside of the process's namespace, so you have to enter it with nsenter.

(I like DynamicUser so I'm glad to find ways to make it easier to use.)

One way to see what processes might have things going on is with lsns(8), which I probably want to use as 'lsns -t mnt'. I don't think there's an easy way to tell whether these have a disconnected /tmp or a merely private /tmp (or perhaps are doing other things with namespaces).

Sidebar: Copying files out of a namespace the hard way

Suppose that you have an entirely locked down systemd service with a disconnected /tmp and that disconnected /tmp contains debugging information you want to copy out. Unless there's a way of accessing a mount namespace from outside of it, this means using nsenter, but that will lock you inside the locked down mount namespace (which may have everything else read-only). The obvious blunt hack to get around this and access the file contents of debugging files is that standard output isn't restricted:

nsenter -t <pid> -m cat /tmp/whatever >/tmp/whatever-out

The shell redirection will be done to the real /tmp since it's set up by the shell before nsenter runs and switches what '/tmp' means. The 'cat' runs inside the namespace with its /tmp switched, so it reads your disconnected debugging file.

(An enterprising person who needed to do this often enough could turn this into a 'nscp' script, perhaps used as 'nscp <pid>:/tmp/whatever /tmp/out'. For bonus points the script could also support copying things in to a namespace, and by extension copying things between them.)

(5 comments.)

Recently on the Fediverse, someone wondered if an out of the box build of Apache (with minimal tuning) could handle 10,000 simultaneous HTTP requests (the old famous 'C10K' target, which definitely was a challenge back in the days). We happen to have a web server that has hit 4,000 simultaneous requests (well, sort of, see later) with every indication that the people (well, programs) that were hitting it would have gone higher than 4,000 if the connection limit was higher. Since that high-water mark we've put an assortment of limits on the web server that cut the connections right down, but if I removed those connection limits in a spirit of experimentation perhaps we'd get to find out how high our Apache setup could go.

(I wouldn't normally experiment with a production service but we don't consider this particular service very important. If it falls over sufficiently often that people start keeping their own copies of the data instead of re-downloading it from us repeatedly, that's a feature.)

The specific configuration of this web server is Ubuntu 24.04 LTS with the standard Ubuntu Apache on a server with 16 GB of RAM, the event MPM, and non-standard settings of:

MaxRequestWorkers   11000
ServerLimit         768

(That's far more processes than we need, but HTTP requests linger around a lot on this server and I was aiming for overkill for a quick test.)

Apache (re)started fine with these settings and is running fine, but I can't say that it stands up to 10,000 connections, at least not yet, because contrary to the previous client behavior (where lots of simultaneous requests would flood in the moment they could), now the number of connections hovers around only 2,000 to 3,000, and sometimes dips lower for an extended time (where 'lower' here is still on the order of 1,500 connections). At one level it's good that people have stopped hammering on this server so much, but at another level it's slightly inconvenient that everyone is being reasonable at the very moment I'd be happy with a stress test.

Also, I have to correct my previous entry when I said we were limiting things to 4,000 connections. Actually, we were tracking and limiting the number of active workers (with a setting of 'MaxRequestWorkers 4000'), rather than the number of connections; I forgot or overlooked that the event MPM allows more connections than you have workers under the right circumstances. When we were hitting the worker limit, it appears that we peaked at around 5,300 simultaneous connections. Presumably the 1,300 or so extra connections were in some state that a worker could handle alongside its other activity.

(The graph I was reading reported the number of workers because that's normally what's important for us; among other things, if Apache runs out of workers it stop answering new requests (including requests to scrape its server status).)

One little lesson I take from this learning experience is that the question of "can Apache handle 10,000 connections" is potentially a little under-specified. At least with the event MPM, there's a potentially big difference between 10,000 active workers and 10,000 connections, a significant number of which aren't taking up a worker. How many such non-worker connections you have may depend on what sort of thing you're serving and how you expect HTTP clients to behave when talking to you (and I suspect this server is on the high side, since people are downloading large files from it).

PS: Since I looked at this in our metrics system, most of the time this particular server has only a "modest" difference between busy workers and current connections with a few hundred more connections. But every so often the difference briefly peaked at a bit over 4,000 of them, perhaps due to Apache shutting down and restarting (or just reloading itself).

Suppose, not entirely hypothetically, that you think you've found a bug in a package and that you have a complicated Emacs environment. If the package is a standard Emacs package, there's generally a simple way to reproduce the problem in a minimal setup; you can do 'emacs -Q' to get a stock Emacs so you can file a nice clean bug report. However, this doesn't work by itself with a third party package that you've installed through list-packages. If you start 'emacs -Q' and try to use the package, you'll probably get an error that your Emacs can't find it (and if it can find the package, you should be suspicious).

This is because the standard Emacs package system stores third partly packages under a directory tree, on Unix typically ~/.emacs.d/elpa, with one directory per package, and none of these package directories are initially on your Emacs Lisp search path. Updating your Emacs Lisp search path is one of the jobs of package initialization; for some time, Emacs has normally done this automatically without needing to invoke anything explicitly (this is the package-enable-at-startup variable, cf). The one exception is in 'emacs -q' and 'emacs -Q', where this isn't done.

Before I started writing this entry I would have confidently given you a recipe for setting up a single third party package. That recipe would have had a lot of old Emacs superstition mixed in, and I don't think it does what I think it did. Now, I'm not sure how you get a completely pure Emacs environment with one third party package and its dependencies, apart from setting up a completely new Emacs environment somehow.

The basics of doing package initialization in 'emacs -Q' appears to be to run either 'package-initialize' or 'package-activate-all'. The former is a command so can be run from M-x after Emacs has started; the latter is a function, so you need to use '(package-activate-all)' in, for example, the *scratch* buffer. Both of these will update the Emacs Lisp load path for all of your packages, but they also both appear to have the effect of activating autoloads for packages, so a lot of the time you don't need to 'require' the package you're interested in. But this also means that the package you're interested in may detect the presence of other packages (through their autoloads) and automatically use them.

To get a completely minimal environment with just your specific package of interest, I believe that what you want to do is the following (in some convenient buffer, such as '*scratch*', and using some convenient way of evaluating this):

(package-initialize t)
(package-activate 'flycheck)

Calling 'package-initialize' this way "initializes" the package system but doesn't activate anything and doesn't add anything to load-path. Then calling 'package-activate' will activate the package and anything it depends on, adding all of the relevant directories to load-path and setting up autoloads and so on. You may then want to "(require 'flycheck)" afterward to fully load the package.

There are two alternate ways that are perhaps somewhat better documented. First, you can add the specific package directory to load-path and then use 'require' to specifically pull in the package. For example:

(add-to-list 'load-path
  "/u/cks/.emacs.d/elpa/flycheck-20260725.1853")
(require 'flycheck)

If the package you're interested in has dependencies, you'll need to add them to the load path too. This has the advantage that you're not touching the package system in any way and you're not going to get surprised by something it does for you.

Another way to do this is to manipulate package-load-list before you trigger package activation in your 'emacs -Q':

(setq package-load-list '((flycheck t)))
(package-initialize)

This appears to initialize only the package you're interested in. I assume that dependencies aren't automatically discovered and activated; if the package has dependencies, then that's up to you to add to package-load-list. Now that I've figured out how to use 'package-activate', I suspect it's not worth bothering with this approach outside of unusual scenarios.

A corollary to all of this is that any time I make a bug report against a third party GNU Emacs package and say that I've reproduced it in a minimal setup, I'm going to write in the bug report how I set up that environment so that people can see if I was doing it wrong. (I thought I was being a little silly when I did it in my initial bug report, but it turns out not at all. Past me made a good call there.)

Sidebar: Startup superstition

My current .emacs has the following stanza:

(require 'package)
(add-to-list 'package-archives
             '("melpa" . "https://melpa.org/packages/"))
(package-initialize)

Based on what I've read in the process of writing this entry, I believe the first and the last bits are now surplus in any modern Emacs and all I need to keep is the MELPA bit. This appears to be a GNU Emacs 27 change (cf).

(2 comments.)

The Apache web server has a quite useful server status page, that I think everyone running Apache should enable (and then protect access to, because it contains potentially sensitive information). If your server ever gets unusually loaded (or overloaded), you can look at the server status page to try to see what's going on, and if you have a general metrics system you can use tools to automatically collect the information it exposes (such as the Apache exporter for Prometheus).

As the Apache documentation covers, the server status page offers this information in two forms, a HTML web page that's intended for people and a 'machine readable' version that presents much of the information in plain text in a relatively easy to parse format. The HTML page comes in two versions, a table based version (the default) and a version not using tables (accessed with the no longer documented '?notable' parameter, and I think it may include some extra information in more verbose format but I haven't looked closely). All of this is great, but there's an unfortunate limitation of the server status page, which is that no version of it provides detailed per-request information in an easily processed form.

The reason you might want to process per-request information is to look for patterns in what IP addresses or network areas have a lot of requests currently, or what URLs and URL areas all (or many) of your requests are for. You can get some of this from the Apache log files, but they don't tell you about concurrency except indirectly (and the log files only tell you about requests after they complete, which can take a long time). When this is happening, you'd like to be able to scrape the server status data into a script that can tell you things like source IP and request distribution from the live data (well, a snapshot of it).

The plain text, machine readable version of the status page doesn't include the normal detailed per-request information, only aggregate data (and the scoreboard), leaving you to parse this information out of the HTML versions. However, neither HTML version makes this easy, because they don't label their HTML with any class or ID information. You're left to know specific information about the structure of the HTML page and where to find the right table and table cells within the HTML. On the good side, this HTML structure is probably not going to change any time soon; my impression is that the output of mod_status is basically frozen for whatever reason; either people don't want to work on it or maybe the current HTML output is considered a de facto API at this point (since people will have written things that parse it).

Even if the HTML and the plain text 'auto' versions of the status page are considered de facto APIs that can't be changed, nothing prevents Apache from adding another parameter and version of the status page that reports per-request information in a machine readable format. But it's probably never going to happen, and even I would consider this a (very) low priority issue for Apache.

(I suspect that there are good, stand-alone programs to parse HTML tables to text format, since that seems like a common need, and maybe programs that extract the per-request information from the Apache server status. But in today's Internet, finding them is another matter.)

Over on the Fediverse, Mike Sheward shared some words of network engineering wisdom:

i’ve said it before and i’ll say it again; you’re not a network engineer unless you’ve moved a ceiling tile to trace a cable and had a small netgear switch fall on your head

You might wonder how on earth you wind up with little switches stuck in the ceiling (well, above the ceiling). At work, we've kind of been in this situation ourselves, so I can tell you our situation, or more exactly the university's situation.

As you might expect, the university has any number of old buildings. When many of these buildings were built or, more likely, renovated for twisted pair Ethernet wiring, wiring and ports and so on were expensive and no one really foresaw how pervasive and popular this 'computer networking' thing would get. The result is that these buildings were often initially wired for a much lower port density than they turned out to need later. In the era of 100 MBit networking, this lead to some significant hacks such as Ethernet splitters, but these hacks can't be used if you need 1G Ethernet (and in practice you do these days).

Renovating a building (or part of it) to add more network wiring and more network ports in the proper, conventional way is expensive and perhaps disruptive (you have to move people out of their existing space to take it apart to run the new wiring through it and so on). At the university, typically this only happens as part of a general renovation, when they have to take the space apart anyway. People have learned to run lots of wiring at that point; the actual Cat-6A cables and network jacks are inexpensive these days compared to the labor costs to pull cable bundles, and you can terminate cables in a wiring patch panel without hooking them up to switches.

If you can't afford to run new wiring, arguably the proper way to solve this is with a collection of small fanless switches in office areas (you want fanless switches because people often object to switch fan noise). However, that's a switch-intensive setup no matter how you arrange it, and it involves leaving switches lying around in the relative open where various things can happen to them. If you don't want to do this, an alternate way of dealing with the problem is to move the necessary switch or switches out of sight by putting them above a false ceiling. Sometimes you can get away with a small fanless switch; at other times, you may need a somewhat bigger and noisier switch, which will be muffled by being on the other side of the ceiling tiles. If you need a few more network wires to support this, there's often ways to run your own network cables yourself back to some area you can put a distribution switch.

(Then you can run cables from the ceiling switch, over to the side of the wall, down the wall (often in a stick-on plastic conduit), and to a network port box that you've used adhesive to stick on the wall. Done carefully this can look almost like real wiring.)

This has some obvious drawbacks and also some less obvious ones. In particular, you or the people who come after you often aren't going to remember that you have those switches up there. When one of them eventually stops working, this can lead to exciting trouble-shooting sessions, and if anyone ever takes the wrong ceiling tiles off, this can result in things falling on them, or at least the sudden appearance of a switch that's being held aloft in mid-air only by the power and network cables connected to it. This can be especially fun if who's in the space has changed since it was wired up with the ceiling switches.

(In theory this can be well documented and the ceiling tile with a switch on the other side can be clearly marked and all of that. In practice my impression is that this is often done somewhat quietly and without such things, partly because some parts of the university want you to do your network wiring properly even if it requires money, time, and so on your department doesn't have.)

(One comment.)

Back when I wrote about using tc to limit the outgoing bandwidth of a web server, I expressed a wish:

For our purposes it would be nice to do something sophisticated to aggregate all HTTPS requests from a single IP address together in a single 'flow' for tc-sfq(8). In theory this is possible with tc-flow(8) but in practice I can't figure out a command line that works right (despite consulting eg tc-sfb(8)'s example).

The good news is that I managed to figure out a syntax that tc would accept. The bad news is that I don't know if it does what I want, because I'm not sure how you see what connections are assigned to what flows.

I'll put forward two variations of tc-flow(8) commands. To start with, I'll set the stage. We start with our bandwidth limited class that will have a tc-sfq(8) qdisc below it, straight from the first entry:

tc class add dev eno1 parent 1: classid 1:10 htb rate 400mbit ceil 500mbit prio 10

Our first option is to do exactly what I said above, making the flow per-IP instead of per four-tuple, and attach this flow classification to our bandwidth limited qdisc.

tc filter add dev eno1 protocol ip parent 1:10 handle 20 flow hash keys src,dst,proto,proto-src divisor 4096

(The 'handle <something>' is apparently critical although I don't know why. I got this wisdom from here after extensive online searches.)

This theoretically makes flows be based on the source IP, destination IP, source port, and the protocol. In our case, for a HTTPS web server with us dealing with outgoing traffic, the source, protocol, and source port are all the same, so this only varies the flow by the destination IP. This is what we want to aggregate all connections from the same IP. Some sources will tell you to use 'perturb N' on this. We don't want to do this because we want flows to stay sticky and not get re-sorted every so often, and we want to set a very large divisor so that we maintain a lot of distinct flows to go with our (initially) large number of connections.

(The tc-sfb(8) manual page has an example like this, basing flows purely on the destination IP, but it's attached to the root qdisc.)

The other option is to be even more aggressive and theoretically group flows by the /24 of the destination IP address, and nothing else (since we're already theoretically only dealing with HTTPS responses sent by our web server, with the protocol, source IP, and source port all constant). This is done with the other sort of flow filter, a 'map' filter instead of a 'hash':

tc filter add dev eno1 protocol ip parent 1:10 handle 20 flow map key dst rshift 8 divisor 4096

If I'm understanding tc-flow(8) correctly, this takes the destination IP and right shifts it 8 bits (then divides the result by 4096, once again to try to have as many distinct flows as we could in theory have simultaneous connections). Dropping the right octet of a full IP address effectively reduces it to its /24. I think we could also use 'dst and 0xffffff00' to get basically the same effect, but maybe that would get more flow collisions when the dust settled.

I believe that we also need to tell our tc-sfq(8) queue how many flows it's supposed to have, matched with the number we used above:

tc qdisc add dev eno1 parent 1:10 handle 10: sfq flows 4096 divisor 4096

Unfortunately, I don't know if all of this works because I don't know of any way to see what flow a given connection is classified into (either before or after tc-sfq(8) gets at it). It could be that my tc-flow(8) usage isn't actually assigning flows the way I want (especially in the second case). It could also be that I need to put my flow filter somewhere else in the tc class hierarchy, or maybe somehow attach it directly to the tc-sfq(8) qdisc.

(Given the example in tc-sfb(8), where the filter is attached directly to the sfb qdisc, possibly I need to attach the flow filters to the sfq qdisc, with 'parent 10:' instead of 'parent 1:10'. This appears to be accepted by tc, although who knows if it's working.)

PS: Based on what 'iftop' is telling me, I'm relatively sure that my flow filter isn't actually working the way I want (with a flow filter either in this version or directly attached to the sfq qdisc). But it's hard to be sure.

I recently shared a war story about how we had load problems on a new web server but not an older one because of a missing firewall rule. In a comment, Aristotle Pagaltzis asked how we'd realized that we had a missing firewall rule. This is a good question, because the sequence of events involves a certain amount of luck and coincidence. So here's that story, which may be a useful example of system administration in action in practice (ie, it's kind of messy).

When we first deployed our new web server for a highly in-demand data set, Apache wasn't configured for very many concurrent connections and was immediately overloaded with HTTP requests, but we mostly shrugged. Because this upset our monitoring system and I did want to monitor the web server at least a bit, I turned up the Apache connection limits to an absurd number. This was mostly enough, and the web server's outgoing bandwidth jumped up to 1G wire rates, which at the time I thought was fine. Soon after that, an apparently unrelated machine began to have NFS performance problems, and after some work we realized that it was on the same 1G switch as the highly active web server and its network traffic was getting crowded out.

In a rush to limit the web server's bandwidth usage so the much more important other machine would stop having NFS problems, I hastily added some tc based bandwidth limits by hand, and by this I mean that I typed 'tc' commands in a shell session. A few days later, we worked out how to make mod_qos based limits work in Apache itself. One of the limits we imposed and fiddled with was a limit on the number of concurrent connections from a single IP address, which had a much bigger effect than I expected. The Apache error log showed that some IPs were hitting it quite a lot, and our metrics system said the number of concurrent Apache requests dropped dramatically afterward (to about half what they'd been before).

After the dust settled from the immediate crisis, we needed to decide if we were going to make the tc-based limits a permanent part of the machine's configuration (making it the first machine officially using tc in production, and we'd have to write something to install them on system startup) or if we'd rely entirely on mod_qos. While considering the tradeoffs involved, I remembered that we had a general purpose 'block brute force things' system on our perimeter firewall, so we should probably make it apply to HTTP and HTTPS requests to our new web server for extra insurance. Immediately after I did this, our metrics system showed another major drop in concurrent Apache connections (shrinking by half again).

(The perimeter firewall's system has a list of IP addresses that it applies the HTTP and HTTPS limits to, and the new web server, with a new IP address, wasn't in the list (until I added it).)

That major drop from the firewall rule was what sparked my realization of what was different that would explain why our main web server mostly hadn't been overwhelmed by this traffic, because that was the only thing out of all of our rate limiting changes that our main web server had in place. Our main web server had no tc limits and we'd turned off mod_qos several years ago and never revisited that change.

PS: We decided to keep both the Apache mod_qos limits and the tc based limits, partly for extra insurance. We've decided that we really don't want this particular web server to run over the bandwidth limits, so having two mechanisms that limits it means they'd both have to fail.

(We're not yet worried enough to look into FreeBSD PF's features for bandwidth limits, partly because if we got it wrong on the firewall, we could affect a lot more than this machine.)

(One comment.)

When I wrote about how early SunOS did diskless workstations, I got a good question in a comment:

Obviously this is slow, but having an indeterminate number of users sharing a circa 1982 mechanical hard drive sounds like a problematic amount of slow. Was this ever truly worth doing? I know hard drives were wildly expensive back then, but surely the work slowdown from having multiple users sharing a single hard drive in this fashion would mean the ROI for those hard drives would seem obvious?

One answer is that diskless workstations were not infrequently used in situations where there was no 'ROI' as such, for example for use by university graduate students (generally not in dedicated offices, unless you were very lucky, but instead in shared in terminal rooms). But in my view, a deeper answer is that there are two usage models of diskless workstations.

In one model, a diskless workstation was an inferior substitute for a workstation with a local disk. It had to do its disk IO over a slow shared 10MBit network connection to a server with mechanical HDDs that were used by multiple people (on all of those diskless workstations), which was obviously much worse performance than a local disk. But you saved the cost of the local disk, and perhaps you bought a low end workstation model (such as the basic Sun 3/50 instead of the better, faster 3/60) because they weren't going to be fast anyway.

In the other model, a diskless workstation was a superior replacement for a serial terminal, in much the same way that X terminals were later. Instead of a single text 'window' and no local computing, you gave people something with multiple windows, graphical capabilities, and some degree of local computing that could be faster than an (over)loaded central server. In the process you might save money on the central server, since it didn't need as much compute capacity as it would if everyone was directly logging in to it (although diskless workstations cost a lot more than serial terminals, so you probably weren't saving money overall).

(The other advantage of the diskless workstation model over the serial terminal model was it was more amenable to incremental upgrades, since you could buy better workstations (perhaps with disks) one by one. It was a "personal computer" model instead of a "terminal" model.)

These two models lead to different calculations of costs and benefits. In the first model, you're losing productivity but saving money on hardware, and the question is how much does the lost productivity actually cost you. You're probably going to give diskless workstations to people who don't have highly valuable productivity. Diskless workstations are a downgrade and workstations with disks will become a status symbol, a sign that you're important enough to call for the extra expense.

In the second model you're gaining productivity at the cost of spending more on hardware. the question is how much extra productivity do people gain compared to the extra cost of diskless workstations over serial terminals (possibly factoring in a cheaper server, fewer serial lines and serial port boards in the server, and so on). In some cases the productivity gains may be significant at relatively modest extra cost. Diskless workstations are an upgrade and a status symbol (compared to serial terminals).

Of course these models cross over somewhere, so you get to look at the relative payoffs and costs of serial terminals, diskless workstations, and workstations with local disks for different groups of people with different productivity payoffs. Once NFS and other shared, writable filesystems entered the picture, things got more complicated because even your 'local disk' workstations might be NFS mounting home directories, shared work areas, and so on, both for collaboration and so that people weren't tied to specific physical workstations.

(When X terminals arrived they added another section to the spectrum, with graphics but without local computation. This could make sense in a variety of ways; some people benefited from graphics but not local computation, and some people needed to do most or all of their compute on the same machine as their data (on HDDs), instead of hauling it back and forth over shared 10MBit networks with the network filesystem protocol of your choice.)

There's likely also a practical commercial aspect to diskless workstations. My impression is that Sun and other early Unix workstation vendors were relatively desperate to get their machines into places (Unix was a new thing, after all), so in the grand tradition of such things they created a low cost entry level version of their product to get their foot in the door, even if it wasn't all that great. If you could initially sell a company some base configuration diskless workstations and a server to go with them for cheap, maybe you could turn that into a later sale of better, more expensive hardware once the company got a taste of Unix.

(Sun would later continue this tradition by selling entry level hardware without hardware floating point.)

(One comment.)

Today, for reasons outside the scope of this entry, we needed to do unscheduled reboots on a number of Ubuntu 26.04 servers that people log in to and use. As is our usual process, we didn't reboot these on the spot; instead we ran 'shutdown -r +NN "<a message about the situation>"' so that people would have a little bit of warning because the impending shutdown would be periodically announced (by systemd, because this is systemd-based these days). Then, to our unpleasant surprise, we discovered that no announcements were happening. Shortly afterward we discovered that the venerable 'wall' program wasn't making announcements either.

Surprisingly, these turn out to be two separate issues. The wall issue is because starting in Debian 13 ('Trixie') and Ubuntu 25.10, the Debian and Ubuntu systemd is built without support for /var/run/utmp (aka /run/utmp), the traditional file recording who is logged in where; wall (which comes from the 'bsdutils' package) only looks in the utmp file. If there's no utmp file, wall is never going to do anything. If you need a wall equivalent, you'll need to write a script that gets the list of active user sessions with ptys and writes a message to them itself.

(For Debian Trixie dropping support for utmp, see eg this debian-devel thread. Apparently one reason for the change is that the utmp format has Y2038 problems. A replacement is available through the wtmpdb package and project, which also gives you a working 'last' command. You have to hook it up in your PAM configuration, and the Ubuntu 26.04 OpenSSH is built without wtmpdb support, so I believe you're going to be missing some information.)

The issue with shutdown not broadcasting messages appears to be because in Ubuntu 26.04 (with systemd 259.5), systemd's logind doesn't know what ttys people's SSH logins are using. You can see this with 'loginctl' or 'loginctl -j', which will have no TTY information for all SSH logins (although if you log in on the console, it will have that). This isn't the case in Ubuntu 24.04 (with systemd 255.4) or Fedora 43 (with systemd 258.9), although both versions are built with UTMP support (in theory this shouldn't matter, since logind's tty tracking is a separate thing). Logind's announcements of impending shutdowns only go to TTYs that it knows about, so since it doesn't know about any SSH login ptys, none of them get any announcements.

Update: Now that I pay attention, Fedora 43's systemd is older than Ubuntu 26.04's. However, Fedora 44 has systemd 259.7 (with UTMP enabled) and its 'loginctl' doesn't have this problem.

If you're using 'who' or 'w' on Ubuntu 26.04 (or at least the version of 'who' from GNU Coreutils, since the uutils version currently suffers from bug #2152801), you might notice that they do report pty information, at least if you have AppArmor disabled (as we do):

; who
cks      sshd pts/0   Jul 20 21:47 (...)
; lsb_release -r
Release:        26.04

This is because while the GNU Coreutils version of 'who' talks to systemd to try to get this information (through a set of systemd library APIs), if there's no TTY information for a session it will also look through /dev/pts to try to find a likely candidate. This works often enough that 'who' typically shows information for most interactive SSH sessions with a pty. The 'w' program, which comes from procps, has a similar fallback if systemd and utmp are both not reporting the tty. Since 'who' and 'w' use different approaches, on Ubuntu 26.04 one may report a tty that the other doesn't.

(Apparently the default Ubuntu 26.04 AppArmor profiles block access by 'who' to /run/systemd/sessions, which the systemd library API uses under the covers to get session information. Amusingly, this only affects 'who', not 'w', as 'w' has no specific AppArmor profile.)

(4 comments.)

I mentioned recently that we'd moved a very popular data set from our main web server to a new one that only handled that data set. On the new web server, we found it necessary to set an absurdly high Apache connection limit of 4,000 concurrent requests, because Apache could run out otherwise (and could even run out at 4,000, it was just less often).

Our main web server has a much lower concurrent connection limit but it didn't experience these problems (and not because we'd imposed connection limits in Apache; we'd turned that off in December of 2022). But as we (I) wrangled with the new web server to get it to stop running out of connections and so on, and even after I got mod_qos working on it, I never paused to ask myself why the new web server had so many problems with this when the main web server had run for years without explosions (or at least very infrequent ones).

(Part of this was because the main web server had started exploding sometimes; that was why we'd moved this data set to its own server. But even those explosions had been less severe than what I was seeing.)

The answer is that for years, our perimeter firewall has had per-IP connection rate limits for HTTP and HTTPS connection to our main web server (among other per-IP connection rate limits, for example for SSH connections). These predate the modern popularity of this data set and were added to stop other abuse, but it turns out that a lot of the connection volume for this data set was coming from a few IPs that were opening up a ton of rapid-fire and often simultaneous connections. Once we applied per-IP limits on both the number of simultaneous connections you could have (in mod_qos) and the rate at which you could make connections (in the firewall), the new web server's connection count dropped like a stone (but people still kept pulling data from it as fast as they could).

In retrospect, the change in the web server's behavior when we moved this data set to a new host was a signal. We'd had only occasional problems on the old web server host one (despite it being actively used for other things) and we had constant ones on the new server (dedicated only to the data set). So we could have asked what was different, and then investigated, and then found the perimeter firewall issue. But on the other hand, this is sort of hindsight bias speaking. Such changes in behavior are a signal, but as system administrators we're drowning in signals and we have to sort out what's meaningful and what's either a coincidence or a consequence of something else (for example, a sudden increase in demand for this data set, which would have also explained why we were suddenly seeing problems even on the main web server).

PS: There's some recent evidence that there was a real but temporary shift in demand for this data set, in large part from people (or software) that make extremely inefficient requests. If these people are done now, or have improved their software, that would be nice. Perhaps all of the connection blocking and limited bandwidth have encouraged them to download things only once and then keep local caches.

(2 comments.)

GNU Coreutils are what they sound like; a GNU version of a bunch of basic, core Unix programs such as 'mkdir', 'head', 'chmod', 'cp', 'mv', and so on. For a long time, talking about 'GNU Coreutils' was unnecessary and you could just talk about 'Coreutils'. Then some people decided to rewrite Coreutils in Rust, the uutils coreutils, which still wouldn't be very important for most people except that Canonical decided to make the Rust versions the default in Ubuntu 25.10 and then 26.04.

In theory the Rust coreutils aim for 100% compatibility with GNU Coreutils and anything to the contrary is a bug. In practice, I found an incompatibility almost immediately when testing 26.04 pre-release, Ubuntu bug reports are useless, and I was pretty certain that other people on our systems would run into other issues, so I decided that we would sit out this round of Canonical making 26.04 LTS people mandatory beta-testers of their current passion project.

Canonical doesn't make it easy to switch from Rust uutils to GNU Coreutils, but they do at least make it possible. The magic apt-get command you need is:

apt-get install coreutils-from-gnu coreutils-from-uutils- --allow-remove-essential

(Taken from here.)

If you do this, I suggest that you immediately do 'apt-mark hold coreutils-from-uutils' (you may not want to hold 'coreutils-from-gnu', since there might be bugfix updates to it for some reason, although the real programs are in the 'gnu-coreutils' package).

The reason you might want to do this is, well, let me quote a Fediverse post of mine:

Ubuntu: you can totally continue to use GNU Coreutils in 26.04 LTS.
Also Ubuntu: build-essential depends on the new Rust coreutils.

Yeah, that's not "you can totally continue to use GNU Coreutils", although it sure is tempting to build my own build-essential package with a different dependency.

What this means in practice is that if you install coreutils-from-gnu, the build-essential package is uninstalled if you have it installed, and if you install build-essential later, coreutils-from-gnu is uninstalled and coreutils-from-uutils (the Rust version) is reinstalled. If you're not paying close attention to all of the messages that an 'apt-get' is printing out, you might miss this (especially if the apt-get is happening in the middle of your general install framework). Then you will be surprised, as I was, when it turns out that your 26.04 systems have Rust coreutils despite you theoretically having switched.

Build-essential itself doesn't do much, although installing it is a convenient way to get some core software building tools (especially if you want to build Ubuntu packages, perhaps to make local changes). What really matters is that 'apt-get build-dep' will insist on installing build-essential, and you may want to do 'apt-get build-dep <some package>' for all sorts of reasons. For example, if you're going to build your own Emacs, 'apt-get build-dep emacs' is a convenient way to get most or all of the development packages it's going to want, rather than looking them up and getting each one yourself.

The build-essential dependency is explicit:

$ apt-cache show build-essential
[...]
Depends: libc6-dev | libc-dev, gcc (>= 4:14.2), g++ (>= 4:14.2), make, dpkg-dev (>= 1.22.11), coreutils-from-uutils

Not 'coreutils' (a meta-package that depends on either), not explicitly 'coreutils-from-uutils | coreutils-from-gnu', a direct, specific dependency on the Rust coreutils. This turns out to be a Canonical bodge from September 2025 that's not in the upstream package (via). Since this is an explicit dependency, dealing with it requires things like building your own version of build-essential that has a fixed dependency (perhaps with dgit).

There may be other packages with specific dependencies on 'coreutils-from-uutils', which is why I suggested you explicitly 'apt-mark hold' it. With the package held (or both coreutils-from-* packages held), 'apt-get install <some package>' will abort rather than flip your Coreutils setup around. Then at least you can find out which new package will make you unhappy with Canonical.

("apt-cache rdepends coreutils-from-uutils' doesn't show me anything on our Ubuntu 26.04 LTS machines, but I don't know if that's complete across the entire Ubuntu package set for 26.04.)

I mentioned recently (when I dug into systemd and your user D-Bus session bus) that some of my machines were set up so that they sometimes started another D-Bus session bus daemon for me. After having done some experimentation I can say that this isn't necessary (or really, proper) and I've now stopped doing it. You might wonder how I got myself into this situation, and that's a story of history.

On my primary desktops, I've never used any graphical login manager like gdm, xdm, or so on (which has long been sort of a heresy), and I've never run a standard desktop; instead I have my own window manager environment. This leaves me having to do a lot of things myself as part of starting the X server and my environment that a standard desktop and login environment takes care of for you.

D-Bus started being a thing in Linux before systemd. In those days, starting your user D-Bus session daemon was part of the jobs of your desktop environment, either internally or through files it put into the standard /etc/X11/xinit/xinitrc.d (where your graphical login manager of choice should pick them up, although I'm not sure how that works these days). Since I didn't have a desktop environment and was doing it all myself, I had to research what was normally run on session startup and duplicate it in my own shell scripts, and one of those things was running dbus-launch with the appropriate arguments. In the way I ran it, dbus-launch unconditionally starts a D-Bus session daemon and sets '$DBUS_SESSION_BUS_ADDRESS' to point to it.

This was fine in the pre-systemd days, when my regular console login didn't have a D-Bus session daemon started for it (or set up to be ready to start). Well, it was mostly fine, because the D-Bus session bus address was in /tmp, and things can happen to files in /tmp under various circumstances. But I think it only very rarely went wrong, enough that I didn't really notice.

When systemd started providing a D-Bus setup of its own, the proper official /etc/X11/xinit/xinitrc.d was changed so that it detected this and stepped out of the way and not started another D-Bus daemon (and desktop environments that did it all themselves internally were changed similarly). But my own scripts never had this in and never noticed, so when I logged in on the console systemd would set up the whole D-Bus stuff for me even though I was logging in on the console and then my xinit based scripts would promptly start another D-Bus daemon and override that.

(Well, systemd set up all of this provided that my session was in the right class.)

All of this shows one of the challenges of having your own desktop environment; it's on you to keep up with this sort of stuff, and you're probably not hooked into the information channels for it (as far as I know, the various desktop environment people talk to each other). There are probably other places where my environment has drifted away from how it should be.

PS: It's possible that I'll run into problems with my switch, because there's one potentially important thing that's different between the two approaches. The old approach started the D-Bus session daemon with a fully initialized environment (since I was starting it from my login shell after logging in), while the new one starts it with whatever minimal environment it gets from 'systemd --user'.

(One comment.)

Recently I was peripherally involved in a Fediverse discussion about (C's) argc and argv (the arguments to your main(), the start of a C program). Famously, argv[] is an array of pointers to your program's arguments (including the nominal name of the program), and it's sort of traditional to terminate it with a NULL pointer (although this isn't required by the Single Unix Standard; its execve() specification is silent on this). If you think about it, having both argc and a NULL-terminated argv is redundant, since you could determine one from the other. So me being me, I wondered how far back argc and argv went in Unix (and if argv was NULL terminated from the beginning). The answer turns out to be that they go all the way back to Research Unix V1, which is before C existed, and argv[] wasn't originally NULL terminated.

Update: Tony Finch pointed out that POSIX actually does specifically require argv[] to be NULL terminated (and the NULL not be counted in argc). See the comment for details.

The V1 exec(2) manual page is specific about both sides of the V1 exec() API (which is expressed in assembly language terms, since C wasn't invented yet). Exec() is called with a NULL-terminated array of pointers to the (zero-terminated) argument strings, but the invoked program receives an explicit count of the arguments along with an array of argument pointers, and the array is not listed as NULL-terminated. The V1 kernel source code for sysexec (in u2.s) doesn't appear to put in a final NULL pointer or any other pointer value after the regular argv[] pointers, so your program has to use argc to know when to stop.

The logic of this split between the exec() API and the API to programs is a bit clearer in the C code of the V4 exec() in sys/ken/sys1.c. Exec() needs to count the number of arguments in order to do things like allocate the correct size of argv[] array on the stack of the new program, and having created that count it might as well pass that to the new program as argc. However, if I'm reading the V4 exec() correctly, it adds a final '-1' right after the normal end of the argv[] array:

while(na--) {
  suword(ap=+2, c);
  do
    subyte(c++, *cp);
  while(*cp++);
}
suword(ap+2, -1);

This trailing -1 remains present all the way through the V6 exec() in sys/ken/sys1.c (and I don't know why it was -1 instead of 0; the V6 crt0.s doesn't seem to make any visible check for it).

Finally, in the V7 exece() in sys/sys.1, we get an actual NULL pointer at the end of argv[]. However, this is less of a terminator and more of a separator, because the addition of environment variables in V7 has turned argv[] into two arrays of pointers stacked on top of each other, one for the arguments and one for the user environment (which is also terminated with a NULL, because otherwise there's no way to tell). Based on how the C program startup libc/csu/crt0.s has a loop, I think that it finds the environment by walking the argv[] array to find the separator NULL, although the kernel is still providing argc as well as the argv[] array.

As far as I can tell, both System III and 4.2 BSD continue to add the separator NULL (it's more obvious in the 4.x BSD source, where there's an explicit copy of '0' into the user stack; in System III, it appears that the user stack section is pre-zeroed so the code just bumps the offset). BSD continued doing this at least as late as 4.3 BSD Reno (cf). Based on this repository, it appears that System V Release 2 for the Vax also separated argv[] and the environment with a NULL (cf vax/os/exec.c).

If there were Unix systems that later changed this to not have a separating NULL between argv[] and the environment (and thus not giving argv[] a terminating NULL), I don't know what they are. Instead, I suspect that either some C compilers on early non-Unix systems omitted the NULL at the end of their (made up) argv or that the ANSI C and POSIX people didn't want to explicitly require it.

Update: See above, POSIX does explicitly require NULL termination.

(Now you know why I was looking at exec() in early Unix and came to understand its argv size limit.)

(2 comments.)

When I wrote up how V7 gave us environment variables, I mentioned that up to V6, exec(2) had a limit of 510 bytes of command line arguments (including argv[0], the nominal name of your program). You can see the check in the V6 kernel exec() code in sys/ken/sys1.c (where it returns E2BIG in this case). You might wonder where this limit comes from and why.

When you exec() something, you discard your current process's memory and address space to create create a new one for the new program. Your current (user) memory includes the argv you're passing to exec(), so the kernel has to copy it from your user space into the kernel and then back, temporarily holding it in some sort of kernel memory. In a modern kernel you might dynamically allocate this kernel memory in exec() through the kernel equivalent of malloc(), but the Research Unix kernels were simple and didn't have that sort of thing. Instead, through Research Unix V6, they got their temporary scratch space for exec() by allocating a disk buffer, reusing a facility the kernel already needed. These disk buffers were 512 bytes long, which is more or less where the 510 byte limit on argument size comes from.

(I don't know why it's 510 bytes instead of 512; I've been unable to follow the code closely enough to see if it slips in a use of the last two bytes of the buffer for something else.)

You might innocently think that using a disk buffer just pushes the problem of dynamic allocation of (kernel) memory back one layer, to the disk buffer system. However, early Research Unix kernels are more brute force than that. The V6 kernel has a fixed (and limited) chunk of memory reserved for disk buffers, the buffers array in sys/dmr/bio.c, with its size set by NBUF in sys/param.h. The default NBUF isn't very large, but early Research Unix ran on small systems and had low limits in general (the same param.h sets a limit of 50 processes for the entire system).

This straightforward approach to exec() and disk buffers goes back to at least Research Unix V4 (I haven't looked earlier than that). In V7, the kernel implementation is rather more complex because it needs to handle environment variables too, but it still sort of uses the disk buffer trick. In order to get the extra space without using much extra RAM, V7 uses swap space, writing to it and reading back from it through V7's general disk buffer system (which probably often meant that the disk buffer you wrote to swap is still in RAM when you read it back shortly afterward as part of setting up the new process's memory). So to copy the exec() and exece() arguments, V7 allocates a disk buffer in swap space, copies from user space to the disk buffer until it fills up, flushes and releases the disk buffer, gets a new disk buffer for the new block of swap, and does it all over again.

(As an extra complication, V7 didn't have page based swapping, it only swapped whole programs. So during an execve(), V7 allocated swap space for a NCARGS sized 'program' and then used as much of it as necessary, one disk buffer at a time. If V7 couldn't allocate the necessary swap space during exece(), it paniced.)

PS: If you look at the V6 code for exec() carefully, you'll see one spot where it does 'suword(ap=+2, c);', which looks odd and wrong. That's because in V6 C, '=+' was how you wrote in-place arithmetic, instead of '+=' in V7 and the C we know today.

These days, a lot of things want you to have a (user) D-Bus session bus (to go with the system-wide one), which will listen for connections on some socket (the bus address). On a systemd based system, the normal D-Bus user session bus socket is /run/user/<uid>/bus, which you can see by logging in and doing, for example, 'echo $DBUS_SESSION_BUS_ADDRESS'. Suppose that you SSH in to some Linux machine that uses systemd, run this, see your expected D-Bus session bus address, and even use 'lsof' to see what's listening to it. Do you actually have a D-Bus session bus active?

Well, maybe, because these days your D-Bus session bus is a socket-activated systemd service. Specifically, it's a user socket and service that's managed by your per-user systemd instance, which is a systemd process running as 'systemd --user' under your uid. When this 'systemd --user' process starts, typically on your first login (including a SSH login), it will start listening on a bunch of sockets, including your standard D-Bus session bus socket, and it will insert a $DBUS_SESSION_BUS_ADDRESS into the 'systemd (user) service manager environment variables' with 'systemctl --user set-environment ...', where many things will then pull it back out (typically including your new SSH login).

Your actual D-Bus session bus and associated processes are only started by systemd if something actually tries to talk to the session bus. This will typically start 'dbus.service', but what that does varies from distribution to distribution. On Fedora, this runs dbus-broker-launch via /etc/systemd/user/dbus.service (which is actually a symlink to /usr/lib/systemd/user/dbus-broker.service), which I believe then starts dbus-daemon itself; on Ubuntu and Debian this directly runs dbus-daemon via /usr/lib/systemd/user/dbus.service. Your distance will likely vary on other distributions.

(The corollary to this is that 'systemctl --user set-environment' and friends aren't using your D-Bus session bus, because systemd does this without starting the the session bus. In Ubuntu 26.04 this communication is done through a systemd socket, /run/user/<uid>/systemd/private, that apparently uses a private API.)

If you SSH in to a server, look at your processes, and there's no dbus-daemon, I believe that you can be pretty sure that you don't have a D-Bus session bus operating yet. One corollary of this is that any surprising delays in logging in and starting your session definitely aren't D-Bus session bus problems, because clearly your session bus hasn't even been started.

(Well, assuming that you can deliberately start your session bus, for example by running 'dbus-monitor --session'. If your session bus refuses to start, that might be your problem. In my case, it's not.)

(This is the kind of thing that I want to write down in case I ever need it again, because I got confused about what was providing my session bus and whether it was activated or not.)

PS: Normally your session bus is shared between all logins, both on the local console and remotely over SSH, but this isn't required. It's possible to start another D-Bus session bus daemon and arrange for that session bus to be used by other processes. However I'm not sure you actually want to do this and it may be a mistake for some of my machines to (still) be set up this way.

(3 comments.)

An exciting change landed in the development version of Prometheus recently, making PromQL arithmetic expressions in time durations a standard feature instead of an experimental one. For me, duration arithmetic expressions by themselves aren't the truly interesting part. What's really exciting is that as part of this change, Prometheus has added some new PromQL functions, especially step().

(Although step() is gated behind an 'experimental PromQL functions' feature flag today, it will be made available as a standard function as part of duration expressions becoming a standard feature.)

For people who are familiar with Grafana, step() is the PromQL version of Grafana's $__interval interpolation variable. When you're in a range query, step() is how big the range step is, which was previously unavailable in PromQL even though Prometheus obviously knew this information. Prometheus has also added range(), which gives you the full size of the range duration (and the *_of() functions for further selection if, for example, the step() might be too small). Since PromQL now allows arithmetic expressions in durations, you can use step() in them, allowing you to write PromQL expressions like 'rate(your_metric[step()])', where the duration will automatically adjust to whatever the range step is.

Where step() is especially handy for me is when I'm doing ad-hoc graphs directly in Prometheus's web query interface (instead of wrestling with Grafana Explore). Previously I had to go through various increasingly elaborate processes to find out what the step value was for a given time range, so I could plug it into rate() or various *_over_time() things or the like. Now I can just ask for 'rate(...[step()])' and it will all work out right, and it will keep working right as I zoom the time scale in or out.

In theory one could replace various Grafana uses of $__interval in PromQL queries with step(). In practice this probably isn't worth it, unless you're running into problems with $__interval for some reason or maybe when you're writing completely new queries for new dashboards and so on. The minor advantage of using step() even in Grafana is that you can easily copy the query out of Grafana and put it directly into Prometheus or a query tool to see exactly what you're getting.

To use step() and friends in Prometheus 3.13, you need to enable some feature flags. But since this is going to be in the next version of Prometheus (unless something goes wrong and the Prometheus developers have to back out this change), I think it's pretty safe to turn on the necessary feature flags and start using step() and friends now, at least in ad-hoc poking around. Even if you have to stop using step() later, it will improve your experience today.

(This elaborates on a Fediverse post of mine.)

Over on the Fediverse, I had a little exchange recently:

[other person in a conversation]: I still haven’t forgiven Sun for NFS. No I’m not bitter.

@cks: It could have been worse, Sun could have stuck with nd.

What I was referencing in my post is a now obscure piece of cursed knowledge that I'm happy to share with you today.

Sun's workstations could boot without a local disk from very early on (because that made them cheaper, not because it made them better), but famously NFS only appeared in SunOS 2.0 (which required Sun to also create the idea of a virtual filesystem switch (VFS), which has appeared in basically every Unix since). The pre-NFS versions of SunOS operated without a local disk by using Sun's 'nd', the 'net(work) disk', which is basically what it sounds like.

SunOS nd(4) was a kernel block device (well, pseudo-device) that did its block IO through the network to the server kernel. As covered in nd(4), the same driver was used on both the client and the server, and the server handles everything in the kernel; nd(8) is only there for server setup purposes. The client wasn't configured with the server's information; instead it found the server through the simple approach of "[it] finds the server by broadcasting the initial request". As you can see from the fact that the manual pages I've linked to are for SunOS 3.0, SunOS kept the nd driver and infrastructure quite a long time after NFS was available (I'm not sure, but it might have only been dropped in SunOS 4).

(You can also see the SunOS 1.0 nd(4).)

By itself, the idea of a network disk device isn't particularly cursed. We used to run iSCSI based fileservers quite happily, there's a general ATA over Ethernet protocol that was at least a lot simpler than iSCSI, and Linux has DRBD (and there's probably others out there). What makes SunOS nd into something special is how it works, which comes from a specific limitation of SunOS covered in nd(4):

One last type of unit is provided for use by the server. These are called local units and are named /dev/ndl∗. The Sun physical disk sector 0 label only provides a limited number of partitions per physical disk (eight). Since this number is small and these partitions have somewhat fixed meanings, the nd driver itself has a subpartitioning capability built-in. This allows the large server physical disk partition (e.g. /dev/xy0g ) to be broken up into any number of diskless client partitions.

What this meant in practice was that on your server, you set up one giant partition and then manually decided on the starting and ending sectors for every nd 'disk' within that partition. Keeping track of all of these and making sure that they didn't overlap was your problem; as the nd(8) manual page dryly notes in the BUGS section, 'no sanity checking of disk partitions is done'.

For extra bonus problems, you might run out of available partitions to use on your server disk because you needed all of the available ones for regular filesystems and your swap area. If you were in this situation you could take the dangerous but necessary step of specifying your network disks using the special 'c' partition (cf dkinfo(8)), which was conventionally used to provide access to the entire disk. This was extra dangerous because you had to make sure that the nd disks you specified weren't overlapping into any regular partitions that you were using, since as nd(8) says, nd itself did no sanity checking. If you said sectors X to Y were network disk X, that's what they were, and goodness help you if some of them were also something else.

(I think this meant you could expose the server's /usr disk partition as a read-only 'public' nd device, so all your diskless clients could mount it, rather than having to put a separate copy of /usr into your nd area. This seems to be explicit considered in nd(4).)

Another charming thing about nd was that it didn't use UDP (or TCP). Instead it uses its own IP datagram protocol because, as covered in nd(4), "IP datagrams were chosen instead of UDP datagrams because only the IP header is checksummed, not the entire packet as in UDP" (and the manual page straight up says it was also done because the kernel internal interfaces were simpler). What this means is that the data sent through nd had no checksum protections on the wire, not even UDP's basic one; you were very much counting on absolutely nothing going wrong on your early 1980s Ethernet network.

(With early 1980s CPUs and so on, it presumably made a real performance difference to not checksum 1024 bytes of data on each packet. Early Sun workstations were not exactly performance powerhouses.)

All of this made using nd extra exciting and somewhat cursed. I don't think anyone really liked it back in the days, and people were happy to move to NFS, which used regular server filesystems and had much less of a chance to blow up your server and your clients in exciting ways.

(5 comments.)

Suppose, not hypothetically, that you have a situation where there's a number of distinct hosts backing some Git repository, such as all of the IP addresses of https.git.savannah.gnu.org, and one or more of them seem to be outdated or not working right. As far as I know, Git itself provides very few tools to examine or control which host the fetching process uses; the best option 'git fetch' has is to select only IPv4 or IPv6 hosts (well, IP addresses).

(Quite reasonably, git fetch's verbosity settings are focused on the Git side of things, not on the network side of things. The network is supposed to just work, or at least fail in an obvious way.)

Fortunately we can take advantage of the simple Git HTTP protocol to directly query every server to see the state of their repository (assuming that they respond). Specifically, we want to use dumb client reference discovery to see the commit ID of one or more references (most often branch heads) on each server. To do this we'll need some way of forcing a HTTPS server name to resolve to a specific IP address, but curl has this feature in the form of its '--resolve' command line option.

(Curl has two ways to remap a HTTP server name; for using a specific IP address, --resolve is easier or at least more obvious than --connect-to.)

So what we want is something like this (assuming we care about the state of the main branch; you can pick another one):

host=https.git.savannah.gnu.org
url=https://$host/git/emacs.git/info/refs
ipv4=$(dig +short a $host.)
# Curl requires IPv6 addresses as
# '[...]'.
ipv6=$(dig +short aaaa $host. |
       sed -e 's/^/[/' -e 's/$/]/')
for i in $ipv4 $ipv6; do
  echo $i:
  curl -sS -L --resolve $host:443:$i $url |
    grep refs/heads/master
done

(I'm using 'dig +short' in this example as the most convenient general way to get the IPv4 and IPv6 addresses of the host, without anything else.)

At the moment, this says that all of the IP addresses are actually responding to Curl and one IPv6 address is outdated (ie, it has a different commit ID for refs/heads/master, and that commit ID is an old one). I will leave a nicer output format as an exercise to the reader; this is a quick hack that I'm writing down in case I ever need it again (and I hope not to).

(One improvement would be a script that you ran in a repository so it could look up the current head commit and only show you mirror hosts that had a different commit ID for their head.)

Actually doing anything with this information is also left as an exercise to the reader. As far as I know, Git doesn't let you not connect to one specific IP address, so you're left with more system level things like blocking connections to the errant mirror host. Right now, I'm just going to remember to use 'git fetch -4 savannah' when fetching from the official GNU Emacs repository (and hope that no IPv4 mirror host goes bad).

(If you're operating mirror hosts you can use this approach to monitor whether all of the hosts are sufficiently up to date and check for persistently out of date hosts. Or you may have a better monitoring method, for example based on internal mirroring data.)

(2 comments.)

Suppose that you (I) write an Apache configuration stanza to make some settings conditional on the module that they're from, so you can enable and disable the module without blowing up your web server configuration (or having to edit it). Your version looks like this:

<IfModule mod_qos>
  QS_LocRequestLimitMatch "^...$" 1000
  QS_SrvMaxConnPerIP 8 100
</IfModule>

Unfortunately, this stanza isn't doing what you (I) think it is, although it looks like it's correct. As covered in the <IfModule> documentation, the name you give IfModule is either a module identifier or a module file name ('the file name of the module, at the time it was compiled'). The 'mod_qos' I've used here turns out to be neither; the correct module file name for mod_qos is 'mod_qos.c' (or at least I think it is), while the module identifier is 'qos_module'.

(As covered in the documentation for LoadModule, you can get the module identifier by looking at the first argument to LoadModule for the particular module. Although I believe that people writing Apache modules can do it differently if they want to, the standard form seems to be <whatever>_module. I don't know if there's any good way to get the module file name other than guessing it's the conventional name of the module (or its .so file) plus '.c'.)

The effect of an <IfModule name> for a name that's neither a module identifier nor a module file name is that you've completely disabled that stanza, since the <IfModule> can never match an enabled module. You might wonder how you can make this mistake, and in my case it's simple. If you started out with an unqualified set of (module) directives and you're adding the <IfModule> with the intention of then (temporarily) disabling the module, well, the module configuration will be ignored after your 'a2dismod' and Apache restart just as if you'd gotten it right. You'll only discover the mistake when you try to enable the module later (or copy the configuration stanza to another web server entirely), and that might be years later.

(In our case we 'temporarily' disabled mod_qos on our web server in December of 2022 and then never re-enabled it because our web server stopped getting obviously overloaded.)

Today I shared a brief war story on the Fediverse:

Today's new and exciting failure mode for a DHCP server handing out leases to dynamic clients: have something on your network that answers pings for absolutely every IP address (yes, it was broken). The ISC DHCP server pings what it thinks is a free IP before handing it out (to be sure), so if something answers all of those pings you have no 'free' IPs on your network and no one gets a dynamic IP.

(Technically this was like a day or two ago.)

The direct symptom of this in your ISC DHCP server logs is some log lines that look like this:

dhcpd[1656384]: Reclaiming abandoned lease 172.17.101.132.
[...]
dhcpd[1656384]: ICMP Echo reply while lease 172.17.101.132 valid.
dhcpd[1656384]: Abandoning IP address 172.17.101.132: pinged before offer

As ISC dhcpd documents (for example in dhcpd.conf's discussion of the 'ping-check' statement), by default dhcpd will ping an IP it's about to dynamically allocate to make sure it's unused. If something answers, dhcpd more or less gives up on the IP address (this doesn't happen for statically assigned IPs, at least according to the dhcpd.conf manual page). The consequence of this is that if you have such a 'screaming' machine, one that's answering ICMP pings for all IP addresses, dhcpd will conclude that your dynamic IP address pool is entirely exhausted and no dynamic client will be able to lease a new IP. For extra fun, apparently some clients will not accept a DHCP IP if there seems to be something else using it.

(I'm not sure what happens when clients are renewing leases.)

Such a screaming machine is obviously broken in some way and you need it off your network, but unless you get lucky, tracking it down may be hard. We were lucky that the machine was using its real MAC to answer all of the pings (which showed up all over the DHCP server's ARP table, among other places) and that MAC was registered with some useful and accurate additional information. Without that we would have been reduced to tracing through switch ARP tables (for switches smart enough to report that) and eventually unplugging sections of this particular network, which would have been pretty disruptive.

This particular network is port isolated, but that doesn't help here. Our DHCP server has to be able to reach the entire network and the entire network reach it, so its ARP requests flood through the network and anyone can answer them (and then its ICMP ping will be routed through the switch fabric to whatever port answered).

(3 comments.)

Suppose, not entirely hypothetically, that you're installing a server with two network ports and during the (Linux) installation, a third network device shows up with a funny name like 'enp1s0f4u1u2c2', which is a USB Ethernet device (despite you not having any such thing plugged in to the server's USB ports). To your further surprise, your server installer can even lease a DHCP IP on this interface, say "169.254.3.1". Congratulations, your server has a BMC, and this BMC probably speaks Redfish, which is sort of the modern, cloud influenced version of IPMI.

One of the things you'd like to do on a server with a BMC is have the server (the 'host') talk directly to the BMC, for example to get sensor information that only the BMC has or to configure the BMC. In the world of IPMI, people had to put together special methods to talk to the BMC, which required kernel drivers, extracting information from SMBIOS, and so on. This isn't the greatest, and is also not at all like how you talk to the management agent in cloud virtual machines, where you generally talk to the management agent by making HTTP requests to a special IP address. IPMI is in part a network protocol, but talking to a BMC using IPMI over the network is completely different from talking to it from the host server.

The IPMI protocol is an essentially custom UDP based thing, which made sense at the time. Redfish instead uses a HTTP REST based approach, partly because by the time Redfish was started, it was obvious that HTTP had become basically the universal protocol (and it was already in use for similar management purposes in cloud environments). So the natural way for a host server to talk to its Redfish based BMC is over some sort of network connection, instead of through some special out of band mechanism the way IPMI does. However, this requires a network interface that's directly connected to the BMC (and nothing else).

You could in theory wire up some sort of semi-virtual PCIe Ethernet device that was connected to the BMC on the other side. But that's complicated. Most BMCs support 'KVM over IP', and as part of that they need to provide virtual keyboard and mouse input, which these days is done by having the BMC present a (virtual) USB keyboard and mouse to the host. Many BMCs can also present USB storage media to the host, for install media. If a BMC is already presenting a bunch of virtual USB devices to the host, the obvious way to provide a network interface to the host for Redfish is through a virtual USB Ethernet device.

(I think all of these virtual USB devices are often presented on a virtual USB hub, and maybe even a virtual PCIe USB controller to go along with the virtual PCIe graphics card and maybe PCIe bridge. There's a lot of funny business that goes on to connect a BMC to the host system, never mind issues like how BMCs may control the host power.)

Since the host needs to have an IP address on this virtual USB Ethernet device to talk to the BMC at the other end, the BMC has a little DHCP server as well as its HTTP server. This internal HTTP server may or may not be the same as the BMC's regular management HTTP server. On some of our servers, this internal BMC HTTP server only answers Redfish requests and doesn't provide the normal BMC web interface that you can get on the BMC's management network interface (which also supports Redfish requests, of course).

(I think this is a sensible security decision on the BMC's part.)

It started with a discovery about my access to the official Emacs repository:

This is my face when my home desktop appears to be persistently talking to one instance of https.git.savannah.gnu.org that is many days out of date and out of sync on the GNU Emacs git repository. Yes, I know, volunteer organization, but how do you even troubleshoot that? At this rate I'm going to have to switch to the Github mirror even though the thought makes me spit reflexively.

(By 'how do you even troubleshoot that', I meant how might I figure out which mirror is out of date and report it. That DNS name has eight IPv4 addresses and eight IPv6 ones, although you can at least restrict Git to either IPv4 or IPv6.)

This led to me wishing for a way to conveniently pull the same branch from two different upstreams. To be specific, the experience I would like looks like this:

$ git status
On branch emacs-31
Your branch is up to date with 'origin/emacs-31'.
[...]
$ git pull
[pulls and updates my emacs-31 local checkout
from some reliable mirror]

$ git pull savannah
[pulls from the official repository and also
updates my emacs-31 local checkout]

As was pointed out to me by several people (once I read the git-pull manual page), you can get almost this experience with plain remotes, but on your non-default remote you have to remember to use a special form, 'git pull savannah emacs-31'. As far as I can see there's no way to tell 'git pull' to do this automatically, since 'git pull' goes from your local branch to the remote (and you can only have one remote).

(At this point you could make a git alias for this specific operation, perhaps called 'git alt-pull'.)

You can also merely fetch from the non-default upstream and then manually trigger the same nominal merge that 'git pull' would:

$ git fetch savannah
$ git merge --ff-only savannah/emacs-31

I also had a pseudo-clever idea that almost certainly won't work, as I can see better now that I've read a bit more about 'git pull':

I could manually edit .git/config so that both the savannah and github remotes updated the same local 'refs/remotes/origin/*' ref(s), but I suspect that this would go badly and also not necessarily ripple through to updating the on-disk state when I did a 'git pull' from the one that isn't listed as 'remote =' for the emacs-31 branch.

Manually switching the remote of the emacs-31 branch back and forth will do that but, well, annoyance.

Given how 'git pull' works, I believe this would give me the same 'you asked to pull ... but didn't specify a branch' error as 'git pull savannah' does in a standard configuration. It's possible that 'git fetch savannah' followed by 'git merge --ff-only' would work (if the shared remote HEAD was updated by my fetch), but that's already two commands and not much different than other options (at the cost of possibly confusing Git).

Manually switching the remote of my local branch back and forth can be done on the command line with git's general '-c <name>=<value>' setting for setting a configuration parameter:

$ git -c branch.emacs-31.remote=savannah status
On branch emacs-31
Your branch is ahead of 'savannah/emacs-31' by 17 commits.

Once again I could make a cover script that set this for all Git commands, or I think I could do it for specific commands through Git aliases (which I think would make it easier to pass command line arguments through compared to a 'git alt-pull' alias).

(The real answer is that I'm likely to switch more or less permanently to one of the mirrors and give up trying to directly fetch from savannah.gnu.org, which is apparently very overloaded and not all that healthy. One reason I cloned directly from savannah is that I had the impression that the mirrors could lag significantly behind, but on a spot check today the Github one was pretty up to date, with commits only an hour or two old. But at least going through this exercise has left me a bit more educated about some Git stuff.)

In a comment on my entry about how sometimes it's the network that's the problem, I was asked what networking workstations typically have on our networks (and if we still use 100 MBit for low speed things). The simple answer is that it's somewhat mixed but mostly 1G Ethernet, often with 1G uplinks from the switches that the office network jacks are connected to. But there's some stories involved.

As a university department, we have been in our current buildings for what is mostly multiple decades, and in some cases since the early 1980s (cf). In the oldest building (with our old machine room), the department's presence predates twisted pair network wiring entirely (there is still disconnected thicknet wiring around our offices); our other most used building predates Cat-6 cabling. As a result, all of this older space is wired with Cat-5 twisted pair and is only good for 1G Ethernet.

We started out using all of this wiring at 100 MBit (including our connection to the university backbone), and even as time went by we had a mixture of 100 MBit and 1G switches and connections. Due to cost reasons and sometimes wiring density problems, we only slowly migrated people's office network jacks from 100 MBit to 1G as we progressively replaced switches in our multi-switch setup. However, we did eventually move everything to 1G, including in our machine rooms (partly because our old 100 MBit switches were getting unreliable). I think most of our '1G' switches in wiring closets and so on are purely 1G, with no 10G uplink, so wired workstations on those 1G network jacks are sharing the uplink bandwidth.

Areas in buildings do get renovated from time to time, including parts of the department's space. For years, these renovations have typically included running new Cat-6 (or Cat-6A) wiring that is properly certified and tested for 10G-T (it's hard to significantly renovate space without needing to tear out the old wiring and then put in replacement wiring). Sometimes the people who are funding the space renovation really do want their desktops to have 10G connections available, and in that case the renovation funding will also include a collection of 10G-T switches for the other end of the office network jacks to plug in to (however many switches needed for however many true 10G office ports, which is often all of them). Otherwise, while we have 10G-T capable wiring in the walls, we only plug our end of the wires (in wiring closets and machine rooms) into 1G switches and everyone gets 1G.

(Newly built out space in new buildings is all wired with Cat-6A, of course, and we try to get funding for 10G-T switches to go with it. There's been some of this over the recent enough past, as the university does acquire a certain amount of new space over time.)

All of this is the theoretical answer. The practical answer is that a lot of people are using laptops (possibly with docking stations) or desktops with wireless built in, and many people default to using the wireless connection even if there's a network jack or three in their office. Typically, the only people who use actual wired connections are those that need their machines on one of our special research group private networks, instead of on our general 'random desktops and laptops' network. In some areas of the department, there's almost no use of wired network jacks because almost everyone is using laptops and the wireless.

(This makes wireless into critical network infrastructure.)

Here in 2026, I wouldn't try to use 100 MBit for anything, even things that don't need 1G, because I wouldn't trust it. In theory your 1G network interfaces and switches still support it (and maybe your 10G ones too, but don't count on it). In practice, it's been a fairly long time since 100 MBit hardware was in common use. Any switches or the like that you still have lying around are old, and these code paths and physical capabilities in network ports aren't likely to be heavily tested. You want to be running your networks in common and well supported setups, and today that means 1G is the minimum speed you should target.

(And if you have cabling that seems to only support 100 MBit, it's probably broken. Network cables do go bad over time.)

(One comment.)

Suppose, not hypothetically, that you have a web server that's using as much of your server's bandwidth as it can get and you would like it to use less bandwidth than that, so that you can get a word in edgewise (for backups, for example) or just because you don't feel like donating 1/10th of your outgoing bandwidth in apparent perpetuity to people who should be building local caches. There are various ways you might do this, for example using FreeBSD pf on your perimeter firewall, but the lowest impact and risk option is to do it on the (Linux) web server itself with tc(8), the Linux traffic control system. Conveniently I've already done a tiny bit with tc to fight bufferbloat latency.

There are probably a variety of ways to do this in tc(8), but what I'm using right now is mostly pulled from the Arch wiki. It goes like this:

1. We have to switch our device, which is eno1 for me, to using Hierarchy Token Bucket as its top level qdisc (queueing discipline), and in the process set a default class that otherwise unclassified packets will be assigned to.

   tc qdisc del dev eno1 root
   tc qdisc add dev eno1 root handle 1: htb default 30 r2q 1000
   

   Following the Arch example, the default class is 1:30 (the root handle of '1' combined with 'default 30'). Because I'm working with high bandwidth, I need to change r2q to make tc happy.

2. Set up a child class to limit bandwidth, here with a limit of more or less half of a 1G Ethernet. Because we're only doing one level bandwidth limiting (ie, we're not sub-dividing it), this can go right under the root parent ('1:').

   tc class add dev eno1 parent 1: classid 1:10 htb rate 400mbit ceil 500mbit prio 10
   

   Pick the bandwidth numbers to taste depending on your irritation (and the interface speed of your server, your outgoing bandwidth in general, and so on).

   The classid is somewhat arbitrary but as the Arch Wiki example shows, there can be a use for setting a numbering hierarchy if you have multiple levels of parent and child classes. In the Arch example, the top level bandwidth limited class is '1:1', then it has child classes '1:10', '1:20', and '1:30' (the default class for traffic).

3. Following the Arch Wiki example, we add a fair queueing qdisc below our bandwidth limit, so traffic flows that fall into this bandwidth limit are (hopefully) a bit better handled. I think this means that one HTTPS reply to a requester on a fast link won't starve all of the others.

   tc qdisc add dev eno1 parent 1:10 handle 10: sfq perturb 10
   

4. Filter outgoing HTTPS traffic into our bandwidth limiting class:

   tc filter add dev eno1 protocol ip parent 1: prio 1 u32 match ip sport 443 0xffff flowid 1:10
   

   This is a u32 match, and it requires some decoding to understand. A u32 match like this is fundamentally 'match <value>/<mask> at <offset>', and if we dump the raw form of this with 'tc filter show dev eno1', we'll get "match 01bb0000/ffff0000 at 20". The 'ip sport 443' format is simply tc-u32's friendlier way of encoding that (well, technically 'ip sport 443 0xffff', since the 0xffff is a load-bearing part of the shorthand). The tc-u32(8) manual page has various cautions about this matching, so I think if you really care you want to use iptables (or nftables) rules to set a firewall mark and then match on that with tc-fw(8).

5. Create our default catch-all class that effectively has no bandwidth limit, and as with our bandwidth limited class, also attach a tc-sfq(8) qdisc below it.

   tc class add dev eno1 parent 1: classid 1:30 htb rate 1gbit ceil 10gbit prio 1
   tc qdisc add dev eno1 parent 1:30 handle 30: sfq perturb 10
   

   We have two choices for 'prio'. The first option is that we can set this to 'prio 10', the same as our bandwidth limited class; in that case, I believe non-limited traffic will share bandwidth with the bandwidth limited HTTPS traffic, basically getting what's left over. Alternately, we can decide that we want non-limited traffic to have priority over bandwidth limited traffic, in which case we want a lower priority so its packets are sent first. That's what we're doing here.

   We need a classful qdisc, but we'd like one that will automatically use all of the bandwidth available to it. I've used tc-htb(8) here because it's what I was already using, but there are probably better options. The bandwidth I put here is simply 'really big', starting with the 1G interface rate.

This appears to work but the tc-htb(8) rate numbers may not correspond exactly to the wire bandwidth you (I) see. For our purposes, this is good enough; we're not trying to limit things to an exact MBytes/sec value, just something that's in the right ballpark.

Now that I've read the tc-u32(8) manual page, I'm pretty certain I wouldn't want to use it for anything significant where I cared deeply about what traffic was getting sorted into what class. Iptables or nftables is going to be much better at matching network traffic, especially in unusual and weird situations, and you can use that in traffic control rules through tc-fw(8). Here it doesn't matter too much if some unusual HTTPS traffic 'leaks' outside of my tc filter, as long as there's not too much of it (plus the whole server has an intrinsic bandwidth limit).

PS: For our purposes it would be nice to do something sophisticated to aggregate all HTTPS requests from a single IP address together in a single 'flow' for tc-sfq(8). In theory this is possible with tc-flow(8) but in practice I can't figure out a command line that works right (despite consulting eg tc-sfb(8)'s example). I'm sure the documentation makes sense to people who have a deep understanding of Linux traffic control, but I'm not such a person.

We've recently been having mysterious problems getting some of our backups to perform well. Also I recently wrote about how we'd wound up with a web server that frequently saturated its outgoing 1G interface with traffic (and it was a feature that it didn't have a faster network link). These two things turn out to not be as unrelated as we'd like, and there's a story or two there.

The problematic backups aren't done through our usual Amanda-based backup system; instead, a particular central machine pulls /var/log and other relevant files from FreeBSD and OpenBSD hosts (either through rsync or direct SSH access) and then writes them over NFS to one of our NFS fileservers (and then they get backed up by the Amanda backups). The problems manifested as terrible NFS performance; the central machine's load average would go to 50 or 70, everything NFS related on it would be slow, and so on. While we like NFS in general we've had quite a share of problems with it (cf, also), so we immediately assumed that this was yet another instance of an NFS problem. First we reduced the IO load from rsyncs in various ways (eg), then we spent quite a while digging at various NFS metrics, both from our metrics system and from the live system during a problem. Nothing really seemed to be the problem; the fileservers had perfectly good performance, the master machine had perfectly terrible actual NFS performance, and especially it didn't seem to be able to write the backup data very fast, in the range of a few MBytes a second.

Since I was pretty sure that the central machine could write to its local disk acceptably fast, I considered switching to a more complex backup scheme where we first rsync'd things to the local disk, packed up this directory tree into a tar archive, then scp'd it to the relevant fileserver, thereby bypassing any NFS write issues. To check that this would work acceptably fast, I started by scp'ing a test file from the central machine to the fileserver. To my surprise, the scp ran at only about 2 MBytes/s. More testing showed that scp's from this central machine to anywhere ran at 2 MBytes/sec at most, which did rather explain the NFS write problems (much like the lack of a CPU explains a server's failure to power on). At this point a penny dropped in my mind.

We're a university department, which means that we don't necessarily have the newest, shiniest stuff around and we keep things in service for a long time. One of the things we've kept in service is basic 1G switches, because quite a lot of servers don't need more than 1G (and as mentioned, sometimes it's a feature that a server only has 1G). But when I say 'basic 1G switches' I mean switches that are so old that all of their ports are 1G, including the port we use for 'uplinking' them into our overall 10G switch fabric (in contrast to modern 1G switches, many of which have one or two SFP+ ports that can run at 10G, or even a 10G-T port or two). This is fine for our normal 1G servers, which don't generate or receive much traffic even in aggregate, but it breaks down badly the moment you put a high volume 1G server on such a switch. For example, our new high volume web server, which was not only saturating its own outgoing 1G interface but was also saturating the 1G uplink from the switch it was connected to, a switch which unfortunately also had this critical central machine connected to it.

There's a programmer saying that if you think you have a compiler bug, you don't, you have a regular bug that you haven't spotted yet. This saying is almost always true, and it generalizes to other areas, like kernel bugs (also) or hardware problems, or networking problems. At our scale, modern networks are reliable, so if we don't have anything obvious wrong (for example, our monitoring system hasn't alerted us that some machine is unexpected at 100 MBit/sec), our network is almost certainly working fine. So for days it didn't occur to us to actually check. Of course the network was working fine, the network is always working fine. It obviously had to be NFS, especially since NFS has been flaky for us in the past. Except that sometimes it is the network and there's even a good explanation for it that becomes obvious once you realize it's the network.

This whole experience has given me some things to think about. On the one hand, years ago I wrote about how an obvious problem isn't necessarily obvious. There are a ton of things that can be wrong and you have to winnow through them somehow. On the other hand, I'm pretty sure that if we'd engaged in systematic troubleshooting from the ground up, we'd have found this pretty early on. For example, the USE method would have had us look at usage, saturation, and critically 'errors', which might well have caused us to look at TCP retransmits on the central machine (which were decidedly high).

Being systematic about troubleshooting is generally a good thing, but at the same time it's tedious. If the problem had really been a NFS problem (as it has been in the past) and I'd followed the USE method from the ground up, I'd have spent a chunk of time verifying that yes, the network was performing fine on both machines (along with the other things I looked at it, like NFS server metrics). Possibly what I should try to do is start out with likely guesses and then when they come up dry (eg, there are no obvious reasons for a NFS performance problem) and I'm getting frustrated, fall back to the USE method or something similar, even though it's possibly tedious.

(2 comments.)

Suppose that you're generating and transferring a file over ssh, for example to create a tar archive of something on a remote server and save it locally:

ssh [...] rem-server 'cd /var/log && tar -cf - .' >/backup/file.tar

If you're experiencing IO problems in this backup process, an interesting question is what buffer size ssh uses for its writes, perhaps because you'd like to make a few large writes to disk (for example, at the natural 128 KByte block size for your ZFS fileservers, even when you're writing over NFS) instead of a bunch of smaller ones.

The ssh manual page doesn't document anything about this, and there's no options to control it in either ssh or ssh_config(5). On Linux, running over TCP from a remote machine, the answer appears to be that ssh will normally do 32 KByte writes (or 64 KByte writes under some circumstances, possibly if the network is fast enough). It's possible ssh will write smaller buffers if the remote command generating the output can't keep up at full network bandwidth, and in general I suspect that there are a lot of things that can change this.

If this is important to you (and I'm not convinced it's important to us), you need to re-buffer the output from ssh in some way. The traditional way is to use dd, but you need to pick the right options to have dd reblock its input. There are other programs floating around but I don't know if any of them are standard.

(Since I looked it up, there's at least the Debian buffer package and mbuffer. There are probably others out there as well that I can't dig up in casual Internet searches.)

Finding out this information requires some way to trace ssh's system call activity. On Linux this is most easily done with 'strace', and you can narrow down all of the system calls that ssh does with 'strace -e trace-fd=5 ...'. You might think that you want to trace file descriptor 1 (standard output), but in fact current versions of OpenSSH ssh rewires its standard output on to file descriptor 5 and make file descriptor 1 point to /dev/null (which can be very confusing when you first encounter it).

(This is one of those things that I look into, don't find much, and then want to write down my negative results anyway for future use.)

Go's maps are famously implemented as hash tables, which is the only reasonable choice. The implementation has gotten somewhat more complicated since I looked at how maps store their values and keys due to the move to swiss tables, and these days you find the comments about how they work in internal/runtime/maps/map.go, but the core is still the same. Recently, a documentation commit landed in the Go development tree that opened my eyes to a bit of subtle complexity I hadn't considered before in Go's map implementation.

One of the things about hash tables is that they hash the value of keys down to some fixed size value in order to do operations more efficiently; in Go's current swiss tables, this is a 64-bit hash. Critically, the hash value of a key must be constant (which can be an issue in languages like Python that let you define a hash function at user level). You also want the actual value of keys to (only) compare equal when they are equal, which can also be a challenge in a language with user defined comparison functions, or just if you're dealing with NaNs.

Go has a quite broad definition of what's allowed as map keys; you can use any type that has == and != comparison operators defined. This includes pointers (which are directly comparable), arrays of pointers, and structs containing pointers (under the rule that a struct is comparable if all its fields are). However, Go pointers aren't guaranteed to be constant values, and today growing and shrinking a goroutine's stack will change some pointer values. This is a potential problem if you're hashing the current integer value of a pointer as part of a Go map key hash; you need that hash value to stay constant.

The documentation commit explains how Go deals with this today, primarily in its comment in map.go. When a Go value is stored as a map key, the Go compiler marks that value as 'escaping', which means that the value will be allocated in the heap instead of on the stack (along with anything it points to). Currently things in the heap never move, so once a key value is heap allocated, any pointers involved have a constant value and the key's hash value will never change.

As the comment notes, this is only done for keys that are getting stored in the map. Keys used for lookup or for delete will never be stored and so don't need to be specifically heap allocated. As the comment also notes:

If we are looking up a pointer which points to the stack, the hash value is ~irrelevant, as the key is guaranteed to not be in the map [...].

(This includes pointers in structs and so on.)

This map key hash stability requirement is a bit of a subtle constraint on any future Go garbage collector that works through copying values around (eg, also). Probably the simplest way to deal with it would be to mark heap pointers involved in map keys and then never copy or otherwise move them. Possibly you could do this on the fly during the garbage collection scanning process, since you need to trace through maps in general to mark their keys and values as used.

(Until I stumbled over this commit message and read into it more, I'd never thought about how the hash table stability requirement might clash with any sort of moving garbage collection mechanism.)

Suppose, not hypothetically, that you use rsync to push an encrypted backup file from the machine it's created on to a fileserver, where it will be backed up by your regular backup system. Because this encrypted backup file is backed up every day, you only need one copy of it on the filesystem, so you use and reuse a fixed name for the file. In other words, we're using rsync somewhat as if it was scp, but with better control over what remote files can be written and (not) read.

When you push (or pull) a file over rsync, rsync normally attempts to optimize what gets transferred by looking for common blocks in the file (how big a 'block' is depends on the file size, or you can fix it with the '--block-size' option, as covered in rsync(1)). This is a nice potential bandwidth saving, but it creates CPU and IO load on both ends as each of them checks their version of the file. In our specific case, we know that there aren't going to be common blocks; since the whole file is encrypted, it's basically random noise. Recently this backup process had some IO load problems, and today in the process of working on this I discovered rsync's '-W' option (also known as '--whole-file'). As the manual page explains, this 'disables rsync's delta-transfer algorithm'; in other words, it stops looking for common pieces between the two versions of the file. Rsync simply sends the whole file (and the receiver writes the whole file).

Since we know that today's encrypted file has no blocks in common with yesterday's encrypted file (well, it had better not if the encryption is working right), '-W' is exactly what we want to stop the receiving rsync daemon from doing unnecessary work (specifically, unnecessary IO). Effectively it turns the 'file copy' part of rsync into scp (although not literally; rsync will normally write the new version of the file to a temporary file and then replace the old version). Now that I know about -W, I'm going to be looking at some of our other uses of rsync to see if we might want to use it more widely.

(For example, we use rsync to back up /var/log from some FreeBSD hosts, and I'm pretty sure that's a good candidate for -W too.)

If you're using '-W', you want to avoid using '--checksum' and instead rely on the default 'quick check' of file size and modification time. This is because using --checksum requires rsync to read and checksum the whole file before the transfer starts (which is something that the manual page warns you about).

One lesson I've taken from today's experience is that when I use rsync, I should think about what I want to optimize (and what can actually be optimized). Rsync's default behavior is to optimize transfer bandwidth, but sometimes you have enough transfer bandwidth and you want to optimize for lower IO, lower CPU, or both (which is sort of our case for this encrypted backup file, with the extra issue that we know rsync can't reduce the transfer size). Alternately, sometimes you really want to squeeze the bandwidth and maybe '-S' and '-z' (and perhaps others) are what you want, even though you'll do more work on both ends.

(It's possible that rsync already has a clever encoding for runs of zero bytes and so '-S' doesn't save you any transfer bandwidth. I haven't tested.)

(This elaborates on a Fediverse post of mine.)

(3 comments.)

There are a variety of ways to automatically or semi-automatically install systems, especially Linux systems and especially over the network. People have built a whole raft of them over the years, often with relatively impressive capabilities. A number of them have significant levels of automation and control, including for things we might want like sophisticated automatic disk setup. Despite that, we're interested in approximately none of them. As a practical matter, we only want to use the standard, native install systems for whatever we're running, which in this case is the Ubuntu server installer. There are two reasons for this.

The obvious reason is that only native installers and install methods are officially supported by the Linux distribution (or whatever other free Unix we're using), and they're also the ones that are most used in practice. The native installers aren't always perfect, but other people work (often quite a lot) on making them work well, we can find plenty of people using them, and so on. If we use another installer, at the very least we're without support from the distribution. We're also likely to be off the beaten path, which means fewer people running into issues before us, fewer bug reports being filed and fixed, and so on.

The less obvious reason is that we do a certain amount of direct manual installation of servers for Ubuntu, FreeBSD, OpenBSD, and so on. The easiest way to do this is to use whatever native installer the distribution ships on their install ISO images. In the case of Ubuntu we customize this a bit, but none of our customizations are essential; we can and have installed systems straight from the main Ubuntu ISO images (and then possibly imported our customizations afterward). We could still do this even if we normally used a third party install system, but less would carry over between the two environments and we'd want to learn and stay familiar with both of them.

These issues aren't absolute blocks on using a third party installer system; we could deal with both of them. But we'd want to be getting something important from such a system, something that was enough of a gain to be worth the extra costs. I can think of situations which would be worth it but they haven't come up so far (at our modest scale and with only Ubuntu really in the picture for semi-automated installs, network installs, and so on).

(This is related to why it took us a relatively long time to build up a network install system, cf. There are specialized automated network install systems for Linux, but they run into this third party issue. Our current network install system for Ubuntu uses our Ubuntu ISO images that we also use for local installs, and that's a feature even though it creates complications.)

Once upon what's now a long while ago, we had a bunch of SunFire X2100 and X2200 1U servers (for example, they were used in our first generation ZFS fileserver). One of the reasons that I loved these servers back in the days was that for free, they came with a full IPMI/BMC setup that supported both KVM over IP and virtual media. I installed any number of servers this way from the comfort of my office, using the KVM over IP as if I was at the console (because I effectively was) and the IPMI virtual media to feed a local ISO image to the server I was installing. In time those servers went the way of all servers (which is to say, into the e-waste dumpster, although it took a while) and none of our later servers gave me that '(re)install from the office' experience with their limited BMCs. For years, we lived with doing in person physical installs and reinstalls of our servers (well, I lived with that, my co-workers don't care as much).

Recently we wound up with a bunch of servers in another building and a need to reinstall them all with Ubuntu 26.04. This pushed me into learning about UEFI network boot, working out how to network boot our customized Ubuntu ISO image, and then the realization that if we were reinstalling an existing, running server we could use a somewhat simpler kexec-based method. This has given us a reinstall experience (and sometimes an install experience) that looks a fair bit like the SunFire X2100 BMC based experience, and can be done from comfort of our office (instead of a noisy machine room a block or two away). My co-workers like it enough that we're talking about using our new network reinstall system even for servers in our main machine room.

On the one hand, it's nice that we can now have this experience with basically any system (it's nicer if it supports network booting, but reinstalls work without that). On the other hand, the experience is a lot more fragile than the BMC-based experience. Our network installs only works because the Ubuntu server installer supports access over SSH, it requires either a running system on the server or a whole collection of network booting infrastructure (including that it be enabled on the server). If the installer blows up (as it does too often in 26.04), you can't necessarily get access to the system or force a power cycle, and you may be stuck needing to go visit the server in person.

(The most practical version of the network install experience also relies on the server having enough RAM to download the ISO image. This seems to require 8 GB of RAM in common situations in my testing, which is far more than our SunFire X2100s normally had.)

This change from hardware support on limited machines to general software that more or less replicates the earlier experience (but imperfectly) feels like a pattern that's happened repeatedly. One advantage of the software version is that it generalizes, for example to virtual machines.

(Virtual machines can have a BMC-like experience, but it's all with a bespoke environment that's specific to that VM system. Network installs are generic across physical servers, virtual machines, and so on.)

PS: Of course the SunFire BMC used a Java applet, which was somewhat painful (cf). Modern BMCs thankfully use straightforward web stuff.

For a long time, I ignored kexec both for my own personal machines and at work. I knew it existed but I never attempted to use it. That changed recently when I realized we could use kexec to start a network (re)install environment without needing our servers to actually have network booting enabled (which they mostly don't currently, for historical reasons). This has led me to do some additional experimentation with kexec.

The most recent experiment was updating to a new Ubuntu kernel and then using 'kexec' to switch to it instead of 'reboot'. On Ubuntu, this can be pretty simple due to some short symlinks, more or less:

kexec -l /boot/vmlinuz --initrd /boot/initrd.img --append "$(cat /proc/cmdline)"
systemctl kexec

But I'm not sure we're going to actually use this in anything except very unusual circumstances.

The good side of using kexec instead of reboot is that you don't have to sit through your BIOS slowly fiddling around with hardware and then GRUB doing GRUB things. This gets your system back on the air anywhere between 'somewhat faster' and 'much faster', depending on how slow your BIOS firmware is (and also your GRUB timeout). Some of our servers have firmware that is pretty close to 'agonizingly slow', which is part of why we often have network booting disabled on them (this often skips a slow BIOS setup process for network cards, even if network booting was low on the boot priority).

But this good side is also the issue with not using reboot, because if you don't use reboot, you haven't actually tested and verified a cold boot situation. You don't know that your new GRUB boot entry for your new kernel works, and you don't know if your firmware is going to discover some hardware problem at an inconvenient time later on, instead of now during a planned downtime when you're available to deal with the situation. Of course probably the GRUB entry works and probably there's not going to be any surprises from the firmware (and maybe you can inspect the GRUB entry yourself to be sure).

For us, verifying the cold boot behavior (and having a simpler and less error prone 'switch to new kernel' process) is almost always going to be more important than a fast reboot. Now that we know about it, we may kexec into new kernels in exceptional circumstances, but I doubt we're ever going to do it routinely.

A while back I wrote about how claiming to be Googlebot is now a bad idea, where I noted that there were (reports of) malicious crawlers out there impersonating Googlebot and other legitimate big crawlers and at the same time, Google and other crawler operators published the IP address ranges their crawlers used. You could put these two together to block these impersonators:

Anything claiming to be Googlebot that is not from a listed Google IP is extremely suspicious and in this day and age of increasing anti-crawler defenses, blocking all 'Googlebot' activity that isn't from one of their listed IP ranges is an obvious thing to do.

After I wrote that entry, I quietly went and added support for blocking crawler impersonators to DWiki, the wiki-engine that powers Wandering Thoughts, and set it up for a few big crawlers with published IP address ranges. When I did this, I didn't expect to block very much, and for months that was indeed what happened; I'd get a few attempts once in a while. Then, this June, the floodgates opened.

For weeks, I've been seeing hundreds of requests a day claiming to be Googlebot (on a few days, thousands of requests). The requests come from a variety of IP addresses at a variety of providers, which I think are mostly or entirely cloud and hosting providers. The top sources by ASN are a rogue's gallery of places that I was already having problems with, like HostRoyale, M247, Latitude.sh, and Web2Objects. But AWS is in the collection as well, and there are a lot of other relatively mainstream providers. Many IPs seem to make only a few requests as Googlebot, and at least some of them immediately retry with another User-Agent value (which also generally doesn't work).

My guess is that this isn't a bunch of different abusive crawlers who've all spontaneously decided to try forging Googlebot to see if it gets them anywhere. Instead, I suspect that this is a large scale campaign by a single abusive crawler, run by people who can afford to obtain a lot of servers at a lot of different hosting providers (or who are prepared to commit various sorts of criminal fraud on a large scale). Ironically, if they'd picked a different tactic, I might not have noticed them among the background radiation of crawl attempts. Forging Googlebot and other known big crawlers is generally sufficiently rare that I actually bother looking at my logs to see it happening.

(I also suspect I'm not the only website this is happening to.)

PS: It feels somewhat ironic that this is happening at the same time as me wondering if I should allow Googlebot at all.

Recently an interesting series of commits landed in Prometheus with the goal of reducing the size of the Prometheus binary by allowing the Go linker to remove more unused code (something it's quite good at in general, although the linker is also deliberately limited in this). The commit with the message that's most informative about what is going on and why is discovery/gce: keep [Google Cloud] Compute SD client from defeating dead-code elimination, and you can read the full details in it. The short version is that if you're using certain sorts of reflection anywhere in your program, the Go linker won't remove exported (public) methods of any concrete type that's reachable through an interface. It doesn't matter how narrow the interface is (it can be the famous and minimal fmt.Stringer); the moment you combine reflection and a concrete type in an interface, the Go linker more or less stops throwing out unused functions and methods. Well, sort of, as the commit explains.

Unlike the standard Go toolchain not doing dead code elimination for package level variables with constant values, this isn't merely the linker deciding it's too much work to do this dead code elimination optimization. Instead it's at least partly a correctness issue. The problem for the Go linker is that reflect allows you to reach through any retained interface value to use any and all exported methods on the underlying concrete type of the value (and any types it contains), using things like Value.MethodByName() and Value.Call(). This makes it hard or impossible for the Go linker to know which exported methods are really dead and can never be reached at runtime.

(This has to apply to concrete types contained in top level concrete types because reflect can reach through structs, channels, maps, arrays, and so on to retrieve underlying types and values, and thus methods on those types.)

The current Go linker is actually doing more work and eliminating more dead code than the documentation requires it to. The documentation for Value.MethodByName() and friends say that they apply to all exported methods (possibly only of a given name), but apparently the linker will skip this for types that are never directly or indirectly boxed into an interface, because such types aren't reachable through reflect. Since all reflect functions that create a Type or a Value take an any (ie, 'interface{}') as their argument, you can't go from a value of a concrete type to either without putting the concrete type in an interface and triggering this. What this means in practice in a program where there's any use of reflect (including in some sub-dependency off in a corner) is that if you put a 'big' type with a lot of direct and indirect exported methods into an interface, all of those methods and all of their dependencies will have to be retained in the binary (and increase its size, possibly a lot), even if you only use a tiny subset of them.

(I believe this includes innocent looking things like merely printing such a 'big' concrete struct, which you might do for debugging purposes or because it has a String() method that does useful stuff. And of course JSON serialization uses interface values; json.Marshall() takes an 'any' as an argument, so there's your interface. While the json package uses reflect internally, it doesn't currently call any of the reflect methods that triggers this linker behavior.)

There are at least two ways around this, visible in the Compute service discovery commit and a similar Kubernetes commit. In the Kubernetes commit, a concrete top level Kubernetes struct was not retained in full in a Prometheus service discovery struct that would then be boxed into an interface; instead, only the methods on the Kubernetes struct that were actually needed were extracted and embedded into a new struct, so the Go linker only had to retained those methods and their code dependencies. In the more complex Compute commit, some processing had to be done dynamically using concrete types that had to be retained, so instead of putting the concrete types in a Prometheus struct (that would then be boxed as an interface inside the Prometheus code), the values of the concrete types were made inaccessible to reflect by putting them inside a function closure, and only the function closure was stored in the Prometheus struct.

One thing I take away from this is that one should avoid using the various reflect method-getting methods if at all possible, both in a program and especially in a package that you expect other people to use. If your package uses these internally, you're creating spooky action at a distance effects on the whole program (and you should probably mention this in your documentation).

PS: The Go linker's dead code elimination is (currently) discussed in general in a big comment in cmd/link/internal/ld/deadcode.go, which is worth reading for some details that I hadn't thought about until now, such as needing to retain all methods that might be reached through interfaces (which is necessary because you might wind up casting an interface value to another interface entirely, eg, also).

PPS: As mentioned in the Prometheus commits, one of the packages that uses reflect this way is go.yaml.in/yaml/v4. For the actual code and usage involved, see here and here, which seem like reasonably sensible uses to me, even if they have awkward consequences.

One answer to how your Ubuntu 26.04 server boots when its root filesystem is on a software RAID array is that it just does and you don't need to think about it. Unfortunately this wasn't the case in the pre-beta version. That's been fixed since (contrary to what I thought until now), but that raises the question of what changed between 24.04 LTS and 26.04 LTS to cause and then fix this.

At some point between 24.04 and 26.04, Canonical switched from booting servers using initramfs-tools (and an initial ramdisk built with them) to booting them with Dracut. Initial ramdisks built with initramfs-tools will automatically assemble all software RAID arrays they see, or possibly anything they see in their embedded /etc/mdadm/mdadm.conf file (it's not clear to me right now). Initramfses using Dracut will normally only assemble software RAID arrays that are explicitly specified with rd.md.uuid= arguments, regardless of what's in their /etc/mdadm.conf. The 26.04 pre-beta server installer didn't add any such rd.md.uuid arguments to the kernel command line, causing the obvious problems.

(At the time our workaround was to create an /etc/default/grub.d file that explicitly added the right rd.md.uuid argument to the kernel arguments. This had the UUID of the root filesystem's software RAID array hard-coded, which is the style of these files.)

Now that 26.04 has been released, if you install a 26.04 server system with a mirrored root filesystem and inspect the resulting server's /proc/cmdline, you'll still find a striking lack of rd.md.uuid parameters (or rd.auto, cf dracut.cmdline(7)). However, if you peek into the initial ramdisk itself, what you'll find is an /etc/cmdline.d/20-mdraid.conf that contains an rd.md.uuid setting. For example:

rd.md.uuid=838c6235:90e967bb:8e64be9f:3c38b0f8

(/etc/cmdline.d in the initramfs also contains a '20-root-dev.conf', at least on a test system. This is partially redundant with the kernel command line itself.)

As covered in the Dracut manual page, when the initramfs boots, Dracut (in the initramfs) will look at /etc/cmdline.d/ and effectively take them as extra Dracut-directed parameters. So apparently the installer has been updated to write this file, fixing the problem without making the fix visible in /proc/cmdline.

(The initramfs also has an /etc/mdadm.conf, which probably opens you up to the traditional issues with it, although some of them may not be an issue any more.)

This 20-mdraid.conf file only exists in the initramfs, not on the booted system, but the overall Ubuntu kernel update system arranges for each new kernel and initramfs to get this 20-mdraid.conf, so your newly installed kernel boots just as your old one did. (I don't know which bit of the overall setup is responsible for this; it might be Dracut itself.)

Many BMCs will tell you what hardware your server or other system has in it. This is a useful, even valuable function, but you shouldn't necessarily trust what your BMC is saying. We recently had a very vivid demonstration of that, where the BMC of a server confidently reported that the hardware had a number of inexpensive NVMe SSDs, a bunch of (expensive) RAM, and a pair of processors, but mysteriously the server wouldn't power on at all. When we opened up the server, we found no NVMe SSDs, no RAM, and especially no processors, which did rather explain the failure to power on.

Since the BMC in question offered us the option of downloading the raw SMBIOS information, my best guess is that the BMC gets its hardware inventory by having the main system's BIOS push the SMBIOS blob to it when the BIOS powers on or otherwise goes through a state change. The last time the system was powered on, it presumably had all of those NVMe SSDs, the RAM, and the processors, and now the BMC has that state latched until the next time the system can power up.

I was going to say that it made sense from a technical perspective too, but the more I think about it the less I'm completely convinced of that. I believe that some of the hardware bits in question report their status over I2C and other separate channels, not over the main system PCIe bus, so potentially they could be queried by the BMC through a direct BMC connection to appropriate I2C busses. It seems even NVMe devices can have separate I2C connections, so they could plausibly be inventoried separately. Whether this can be done with trickle power with the system powered off is another question. Perhaps what's going on is that the BMC can get hardware information independent of the system processors, but only when the system is powered up enough to supply power to everything, and that can't happen with no processors.

(This also shows vividly how a BMC can be a completely separate thing from the main system. There's no processors or RAM in the server, but the BMC is perfectly fine. It would be nice if the BMC could notice this and tell us, but I suppose this is a quite unusual situation that's not worth adding extra hardware or software for.)

Looking back, this isn't the only time I've seen this sort of BMC behavior. I've seen more than one type of server where some of the hardware sensors reported through IPMI over the management network were only available when the main system was powered on. In the case of sensors this was obvious because they reported no value when the main system was powered off. In the case of our hardware inventory, presumably the vendor decided that the value of the BMC reporting something when the main system was powered off outweighed the problem of it being potentially out of date.

PS: The interesting thing on top of all of this is that the BMC offered to let me configure the server's BIOS (through an entirely separate area of the BMC's web site, which switched to HTTP Basic Authentication). That's a neat trick when you have no processors, but maybe the BMC has software (and hardware) that can write to the NVRAM where the BIOS stores those parameters. It's also a bit disturbing when part of the BIOS parameters are processor dependent, but probably that's using the same BMC-cached information as the hardware inventory.

Suppose, unfortunately not hypothetically, that you have some machines that the Ubuntu 26.04 LTS installer kernel sometimes gets a kernel oops during a network based reinstall (we don't think this is a hardware flaw, but who knows; these machines were stable on 24.04). Further suppose that you're not network booting these machines but instead you're using kexec to boot them into the installer environment. This creates an awkward situation, where the over the network installer may have gotten far enough before the panic to have written over enough of the disk (with the previous install on it) so you can't reboot from it. Fortunately there is a way out, because you can contrive to kexec the installer environment again from within the installer environment.

(A kernel oops is not a kernel panic, and normally doesn't reboot the system unless you've made specific settings changes, which the Ubuntu 26.04 LTS installer environment hasn't done. In this case, that's a good thing for us.)

Once you've used 'kexec -l' to load the installer kernel and initrd with the required command line arguments, you have two choices for actually rebooting into the kexec kernel. You can use the plain and normal 'systemctl kexec', which will "shut down and reboot the system via kexec" (and is what you may have used to boot into the installer from the running system), or you can directly use 'kexec -e'. Based on our recent experiences, if you're rebooting the system because the kernel hit an oops, you probably want to use 'kexec -e', not 'systemctl kexec'.

The problem with a plain 'systemctl kexec' is that as the manual page tells you, it will try to go through the usual standard orderly shutdown process. If your system has hit a kernel oops, this process may not finish. Not even 'systemctl kexec --force' will necessarily finish, because that still tries to do some things in an orderly way. By contrast. 'kexec -e' proceeds immediately to the new kernel (as I believe 'systemctl kexec --force --force' also does, cf). Abruptly rebooting into the new kernel is not normally what you want to do when you're going to use things on the disk and so on later, but when you're already in the middle of the network installer environment, everything is going to get thrown away anyway and an orderly shutdown is pointless (and in this case, dangerous).

(An orderly shutdown at the end of the install process isn't pointless because the installer needs to finalize various things.)

We'll probably keep using 'systemctl kexec' when we're using kexec to go from an existing local install to booting the install environment over the network. It's always possible we'll change our mind and go back to the regular system (perhaps I made a mistake), and in that case an orderly shutdown is better.

One of my standard little scripts is something I call sshup, which waits for a machine to be 'up' by periodically checking to see if its SSH port is responding. As mentioned in my original entry on sshup, I actually have two versions of this script and recently I discovered that the difference is quietly important.

One version of the script uses Netcat, on our Ubuntu machines. The specific Netcat command line it uses is:

nc -w3 -q3 -z "$1" ssh >/dev/null 2>&1

(I didn't used to need the redirection, but Debian reverted a patch and Ubuntu 24.04 picked up the change.)

The technical effect of this is to try to connect to the SSH TCP port and exit once it's done that (or been rejected or timed out), with an appropriate exit status.

The other version doesn't use Netcat, but in Netcat terms what it does is the equivalent of:

nc -N "$1" ssh </dev/null >/dev/null >2&1

The technical effect of this is to try to connect to the server (if possible), immediately shut down the sending side of the TCP conversation, and wait until the server closes the connection completely.

If the server is working properly, these two versions have the same result. But if the server isn't working, the answer these give is different, with the second version being more useful. The problem with the first version is that it only checks if the kernel is willing to let you make a TCP connection to the SSH port; it doesn't check if the SSH daemon is actually responding (normally, the SSH daemon will print a banner, then see that there's nothing more to read from the network and close the connection). If the server is broken sufficiently so that the kernel will accept the connection but the SSH daemon won't run, the first version will tell me that the machine is up and the second one will correctly tell me I can't SSH in to the machine.

(Recently a server hung during shutdown in exactly this way, which is a story for another entry.)

This difference is also relevant for health and monitoring checks for services (and famously so). Connecting to a TCP port is the very basic step in a health check; after that is checking that you received a banner or a canned response from a service (for HTTP services, perhaps a '200 Okay' from a health check endpoint), and after that is checking that you can do something meaningful that's part of the service's regular activity. Of course the higher level you go in checking the service, the more specific your health check is to the service (whether you implement the check in code or in a configuration file that says what to look for). But at the same time, it's more meaningful because it comes closer and closer to what actually matters.

(But there's a tradeoff in how close you come versus how much work and so on it is. You make that tradeoff for individual services based partly on your knowledge of how that service works and what's likely to fail. For my 'is a machine up' checking script, seeing the SSH server banner is good enough basically all of the time. And if my sshup script says the machine is up but then I can't actually log in, that's telling me something useful too.)

(3 comments.)

Recently I wrote an entry about a simple but difficult wish I had for a certain sort of terminal pager that handled emoji. In comments, people suggested that the venerable less would do what I wanted, which was something I'd already tried and discovered it had behavior I didn't want. Except, well, let me quote my eventual comment:

It turns out that my testing of less's behavior was being contaminated by my usual $LESS settings, and I think that 'less -XRn' does more or less what I want (I was being fooled partly because my usual $LESS includes 'c', well, technically 'C').

I've been using less for a long time and for most of that time I've had the same set of less settings, burned into a $LESS environment variable set in my dotfiles, and also the same set of alternate less key bindings (where N and P move through files instead of their normal actions). I had long since forgotten the specifics of my $LESS settings when I tested less; my settings were just how less worked. And so I was fooled into initially thinking that less didn't work for what I wanted.

(I remember N and P partly because when I use less as root or on a machine I haven't particularly set up my account on, I don't have them and I miss them.)

My $LESS setting is far from the only environment variable I have set in my dotfiles. Without looking through them carefully I couldn't tell you what I have set, why I set them that way, and whether or not they're still correct or useful. The same is true for the configuration files for various programs, where I'm sure that surprising things lurk.

(This isn't the first time I've had old personal dotfiles cause problems.)

This is of course just the personal manifestation of a general problem we have in system administration. We set up a lot of settings and configurations that make sense and work at the time, and then we forget about them because they're in the background, just doing what we expect. They can readily become our mental image of how the program or system behaves, because it's certainly how it behaves for us. Then someday either they clash with what we want to do (as my standard $LESS did) or they don't work any more because things changed out from underneath them (cf).

I don't have any good answers to this. Some people will advocate not changing any settings from the defaults, but not only is that giving yourself countless little papercuts over time, it doesn't even necessarily protect you; programs can change their default behavior so something you've become accustomed to doesn't work any more.

(It's good to document the settings and why you made them, but that only helps with half of the problem. It's still easy to forget that you even have custom settings. Should you budget an hour every six months or whatever to look over and check your settings? Well, you can put it on the calendar, but I don't think it's going to work in practice, not after the first few times, because it's boring.)

(2 comments.)

One of the broadly accepted things among at least system administrators is that our own code is a liability (some programmers may resist this idea somewhat more). If your systems run on a collection of locally developed, bespoke programs, scripts, and so on, someone has to maintain and update all of that (I sort of wrote about this long ago, also). If you use standard programs instead (or at least as much standard code as possible), hopefully someone else does that, and you can choose which programs and what code to use based on how well maintained it seems to be at the moment.

(This breaks down when you're large enough, but I'm talking about more modest sized organizations or groups.)

It's recently struck me that the obvious extension of this is that configuration and configuration files are also a cost and a liability (partly as a result of thinking about the staff time cost of our email system). This is obvious when a configuration file embeds what is effectively code, such as Prometheus alerting rules (which have a whole array of clever tricks that you wind up learning), because code in any form is still code. But it goes beyond such embedded code into any configuration that you have to maintain. If you need a complex Bind configuration file or a complex DHCP setup or a tangled Apache configuration (perhaps for a very special server), some day they're probably going to have to be changed or at least understood again.

We can't operate with no configuration and we shouldn't try to. Under the right circumstances, code and configuration are both more of an asset than a liability. But I think we should be cautious about it. Just because we can create a complex configuration that does a lot doesn't mean that we should. It's tempting to create a complex configuration that does lots of things to meet your needs and support all the features you want, and at least for me there's an appealing problem solving aspect to it. But the result can be something that has long term costs and consequences.

Have I created some overly complicated configurations at work? Probably yes. These configurations have given us useful or important features, but they've also saddled us with complexity that we're now stuck with because we (the sysadmins) and other people have come to depend on the features the configurations create.

(Ultimately, one issue is that as a system administrator, it's hard to persuade myself to leave a problem unsolved. Simpler configurations usually mean living without things and we often put in the things for good reason, so simple configurations would require telling more people "you can't have that because we don't feel it's a good idea to support it, although we could".)

(3 comments.)

Ever since I started using PyPy and then had it quietly work fine for years, I've been keeping it in mind as a generally easy way to speed up any Python program that could benefit from a performance boost (which programs PyPy could accelerate sometimes surprised me). But that was in the era when most everything we had was Python 2 based. Now that I'm moving more and more things to Python 3, there's a little issue opening up that I've been thinking about. That issue is what versions of (C)Python PyPy supports, which is to say which language version it implements.

If you look at the PyPy site, you'll see that of Python 3 versions they support, the most recent one is 3.11, which the regular ('CPython') version of was released in late 2022. If you've followed a bit of PyPy development news, you may have read that the PyPy project is resource constrained and thus lags behind CPython for things like implementing language features or standard library changes. My understanding is that PyPy's timeline for implementing language features beyond 3.11 is "whenever we can but don't hold your breath".

On the one hand, Python 3.11 is a perfectly okay version of Python. On the other hand, Python 3 keeps adding more features in newer versions, including useful features. One area I've definitely noticed some changes is in type annotations and typing in general. Typing changes the code I write and I'm fairly certain that the sort of type hints that are (conveniently) available probably will do so too. And at a certain point, more and more code that I want to work with will require something more recent than Python 3.11. If I'm lucky, I'll get warnings about this; if I'm not, I'll some day discover that the reason pipx isn't upgrading various things I have installed through it is that the newer versions aren't compatible with Python 3.11.

(For all I know this is already happening. Or maybe pipx explicitly complains about this situation. Although I have some machines not using PyPy with pipx, and so far they have the same version of programs I install everywhere, such as Python LSP servers.)

In the long run I suspect that this is going to lead me to abandon my use of PyPy with pipx, and probably to step away from PyPy as Ubuntu LTS Python versions keep increasing beyond what PyPy supports. Right now, we have Ubuntu 22.04, Ubuntu 24.04, and Ubuntu 26.04 machines, which have Python 3.10, 3.12, and 3.14 respectively. PyPy's 3.11 doesn't look particularly bad (and it will be years before we stop having 22.04 machines). And of course Ubuntu 28.04 is a fairly long way away, and PyPy might increase its Python 3 version by then.

(This makes me feel a bit sad. PyPy has been good for me, and it's still a great option for Python 2, one that we may turn to sooner or later.)

A while back on the Fediverse, I mentioned something about our departmental web server:

Wow, people do some extremely slow downloads from work's main web server. We roll logs at midnight and Apache wrote the last log record to the old file at 03:58 (for a request that had started at 23:27).

(For our sins we've become a load-bearing source of ML image training data, for the "CIFAR" image dataset. We average 40 Mbytes/sec outgoing web server bandwidth most days.)

(This also got mentioned in passing in my entry on 24 hours of that server's logs. I call us 'load-bearing' not only because of how popular this dataset is but also because in the past the department has gotten plaintive emails from people about it when the dataset wasn't accessible.)

Recently, so many requests started arriving that they overwhelmed our main departmental web server and we decided to deal with the problem by moving this data to another physical machine with its own, dedicated web server (and then using HTTP redirections to push 'people', which is to say software, to fetch the data from that new web server).

(As is so often the case at universities, this isn't an officially published dataset that had been given a permanent URL. Instead, it's part of the home page of a now-departed graduate student. All of this home page data normally lives in a special set of filesystems on our NFS fileservers, but this dataset was so popular that the web server's kernel always had it cached in RAM.)

Now that the data is on its own server (which is still running Apache, because we use Apache for everything), this produced some interesting data, to wit:

How our role as a load-bearing source of ML image training data is going: we had to move the data to a special web server just for it, and that web server has been saturating its 1G network link since midnight local time. It has almost 4,000 connections (the maximum currently allowed).

In something that vaguely amazes me (because I'm sometimes stuck in the past), the machine is using under 10% CPU despite pushing wire rate TLS.

(That's 10% CPU on 4 core HT machine, so out of 8 nominal CPUs. It's a fairly old machine, a Dell R230.)

To be clear, that 4,000 connection limit isn't really a sensible one and you wouldn't normally do this for this sort of bulk server in our situation. At 4,000 connections, if we assume they're all downloading large multi-megabyte files (typically they are) and the available bandwidth is split evenly, each request is getting about 30 KBytes/sec of bandwidth. The reason we set such a high connection limit is because as far as I know, in Apache there's no way to reserve a certain number of connections for use by certain IP addresses, such as your monitoring system. We have a 4,000 connection limit to make it as unlikely as possible that our monitoring system is frozen out and triggers alarms.

Naturally people are not getting their data very fast from our server, but that's not our problem. The current server stats say that since restart Tuesday morning the average reply size is 14.9 MBytes and the average reply duration is a bit under four and a half minutes, which is an extremely unimpressive data rate.

People in that current 4k requests thundering herd are getting their data even slower, of course.

These are Apache's built in statistics, which report the state since server start. This server doesn't always have that many requests and run at wire saturation, although it's happened twice since Monday morning (and the first time was sustained for close to 32 hours). At quieter times the server can run at 'only' 70 to 80 MBytes/sec of outgoing traffic.

You might be surprised that such a popular server only has 1G networking. Well, there's a good reason for that:

Could we put a 10G-T network card in the current server for this dataset? Sure. And then we'd probably blow out all of our outgoing bandwidth to the university backbone. So no. That this server only has a 1G interface is a feature, not a bug.

There are a whole host of ways to limit overall server bandwidth, but the absolutely simplest and most foolproof way is to put the server on a limited-speed network connection. With a 1G network interface, this server is physically incapable of generating more than about 120 MBytes/sec of outgoing traffic. There are some side effects (logging in to the server and interacting with it is quite slow when it's running at saturation like this), but we can live with them.

(There's 2.5G and 5G networking these days, but we don't have the applicable switches and network cards, and given why this 1G saturation traffic seems to happen, I'm not in any rush to increase this server over 1G.)

Sidebar: Where the traffic comes from

As of about 19:40 when I took the stats, there had been 332,000 requests started since midnight local time, from 13,055 different IP addresses. The most prolific single requester is a single Google Cloud Platform IP that made 8,200 requests and probably received about 141 gigabytes of response data. This is far more data than the CIFAR dataset contains, so this IP address was clearly re-fetching data it had already fetched. Unfortunately I believe GCP doesn't charge for ingress traffic. A cluster of Azure IPs in the same /24 made thousands of requests each and the subnet received an aggregate of 1.8 TBytes of responses.

My suspicion is that these are processing jobs or training runs or the like being launched and re-launched repeatedly, with no effective local caching of the fetched data (either because the software doesn't cache at all or because everything is being done one at a time on ephemeral setups, so the just fetched and cached data is immediately discarded). In the past I've seen a similar pattern of high volume re-fetching from what was clearly a HPC cluster at a university.

Overall, 36.25% of the requests are from Azure, 9.89% from GCP, 6.78% from Alibaba IP address space (AS45102), and then everything is under 4% and I'm not going to bother writing it out.

(2 comments.)

In theory, network booting with UEFI is fairly straightforward. Unlike the earlier BIOS based network booting, UEFI defines network booting in the standard itself (although it's not required). In practice, vendors of x86 servers have found any number of creative ways to put stumbling blocks in your way in their firmware (aka 'BIOS'). Here's an incomplete list of reasons that your server might not be willing to do a UEFI network boot the way you expect it.

• The firmware might not even be in UEFI mode, in which case at best it's trying to do BIOS PXE boot, which requires a different set of support infrastructure on your boot server that may not be there (cf, also).

• There's no UEFI network booting option (or options) in the server's UEFI boot order.
• The UEFI network boot option is trying to boot from the wrong interface (on a server with multiple interfaces).
• The firmware only supports UEFI network booting using the built-in 1G network interfaces, not with your 10G-T add-in PCIe card that is actually connected to your network.

• UEFI network booting is enabled in the boot order, but your firmware settings have disabled network booting for all network devices that the firmware will try. Depending on the firmware, this might be a master option that applies to all onboard network interfaces or it might be a per-interface setting.

  (In some firmware this isn't even in the regular 'BIOS' setup screens, but is instead in a completely separate top level configuration system for (some) network devices.)

• Your firmware settings have the network devices in 'legacy' network boot mode (ie, PXE booting) despite your firmware being in UEFI mode, so the firmware won't touch them for UEFI network booting. Sometimes the firmware's setup screens will vaguely describe this as the device firmware being in 'legacy' or 'UEFI' modes, and not specifically mention network booting.

  You might ask why vendors would allow this configuration mismatch. I have no answer, I just know that at least one vendor does have UEFI firmware that will let you do this.

If the server's firmware has a specific 'PXE/UEFI network boot' option that it offers to you on startup (along with things like 'enter setup' or a regular boot menu), this boot option may or may not provide any useful information if the firmware decides it has no valid UEFI network device to boot from. You may be in a situation where you pound on F12 (or whatever key is applicable) but nothing happens and you drop through to whatever 'boot failed' option the firmware has available.

(If the firmware is feeling especially hostile, it will drop you into the UEFI shell [PDF], and see also the Arch wiki page.)

Similarly, if the server offers you a general one time boot menu, but when you pick it there's no UEFI network boot options, this could mean either that the general UEFI boot ordering in the firmware doesn't have network booting included or that the firmware thinks there's no valid UEFI network devices.

There are typically at least two ways to disable (UEFI) network booting on servers: you can take it out of the boot order, or you can tell the often built in network devices to not do network booting. If you've historically disabled network booting on your servers because you don't use it and typically all it did was add extra time to server boots as the firmware poked at network devices, you may have done either or both of these and now need to reverse either or both.

(In non-UEFI, BIOS MBR boot mode, network device firmware has historically fiddled around with PXE stuff even if the BIOS boot sequence wasn't going to wind up trying PXE booting. Or at least it did on some of our servers, so we often turned that off.)

All of this can leave you hunting all over your server firmware's setup screens in order to find whatever magic bit that's making the system not boot from the network the way you want it to. Good luck, but start by checking UEFI boot order and (network) device settings.

Yesterday I wrote an entry about why servers running Ubuntu can stall on boot for two minutes; the short version is that in 26.04, a network interface that either has no carrier or that has nothing else on it will cause systemd-networkd-wait-online to wait for two minutes in the hopes that this changes and the interface becomes healthy. My core diagnosis of the trigger for the problem was correct, but I had the wrong fix because in my testing, I made a classic mistake that's only really possible with virtual networking for virtual machines. There's an often invisible difference between an isolated virtual network interface and one that has no carrier, and I tested with an isolated interface, which stalls under some circumstances and fixed by some things, but not a virtual machine network interface with no carrier. The latter requires a more thorough fix.

I will put the answer at the front: you need to set every disconnected interface in your Netplan configuration to 'optional: true'. Setting 'ignore-carrier: true' isn't necessarily sufficient unless the interface has carrier. If you're mangling your 26.04 Netplan configuration anyway, I suggest you also set 'accept-ra: false' on every inactive interface (and maybe your active one too, unless you use IPv6 RAs). However, you should not set 'optional: true' on your main interface.

There are an assortment of things that will cause systemd-networkd-wait-online to think that it should wait until a particular interface is configured. A probably incomplete list of them is having a static IP address specified for an interface, having DHCPv4 or DHCPv6 enabled on the interface, and having IPv6AcceptRA=yes (to my surprise, although this is sort of covered in the documentation). Under normal circumstances, systemd-networkd will not do any of these things (or attempt them, for DHCP) if the interface has no carrier, which will cause the interface to not be configured. So it sounds like what you want is ConfigureWithoutCarrier=yes, so systemd-networkd configures the interface even without carrier and systemd-networkd-wait-online is happy. Unfortunately this has an important limitation spelled out in its documentation:

With this enabled, to make the interface enter the "configured" state, which is required to make systemd-networkd-wait-online work properly for the interface, all dynamic address configuration mechanisms like DHCP= and IPv6AcceptRA= (which is enabled by default in most cases) need to be disabled. [...]

(Although it's not really clear from the documentation, as far as I can see IPv6AcceptRA=yes is probably the default in many environments, as this manual page excerpt says.)

To disable this behavior when you have ConfigureWithoutCarrier=yes, in theory I believe you also need to set RequiredForOnline=no on the interface (or disable DHCP, IPv6 RA, and so on). Well, if you're working in a purely systemd-networkd world, which we're not on Ubuntu. Instead we're working in the world of Canonical Netplan.

Netplan doesn't simply run systemd-networkd-wait-online and let it decide what to do and what interfaces to consider. Instead, Netplan writes a drop in configuration file, /run/systemd/generator.late/systemd-networkd-wait-online.service.d/10-netplan.conf, and that configuration file tells systemd-networkd-wait-online specifically what interfaces to wait for and what (minimum) states they can be in; this minimum state is normally 'degraded' (as reported by networkctl. As far as I can see, Netplan will normally list and wait for every network interface that is mentioned in your Netplan configuration and that is not set as 'optional: yes'. Also, systemd-networkd-wait-online specifically waits for all of these interfaces to not be in the 'configuring' setup state (again, as reported by networkctl, and as more or less documented in its manual page).

If an interface has no carrier, it's permanently stuck in networkctl's 'no-carrier' state and will never reach the 'degraded' state. If this interface is listed in your Netplan configuration, Netplan's systemd-networkd-wait-online configuration file will say it has to reach 'degraded', and so systemd-networkd-wait-online will time out. This happens regardless of what other Netplan properties you have set for that no-carrier interface; if it's mentioned at all and is not 'optional: yes', you will always have a two minute timeout on boot. This includes an interface with just a do-nothing property like 'renderer: networkd'.

If you specify 'ignore-carrier: true' on an interface, Netplan will duly write ConfigureWithoutCarrier=yes to the applicable .network file. If an interface actually has carrier, this will make an interface with 'accept-ra: true' (or no mention of it) finish configuration immediately regardless of whether or not something responds; the interface will be "degraded" but also "configured" and the systemd-networkd-wait-online configuration Netplan has created will consider it ready. This is unlike the standard behavior covered in the manual page quote above, because Netplan likes to surprise people.

(If DHCP is on for the interface, systemd-networkd won't complete configuration until and unless it gets a DHCP answer, and systemd-networkd-wait-online will have to time out.)

Setting 'ignore-carrier: true' on an interface with a static IP in Netplan will allow that static IP to be configured even when there is no carrier, but it won't make that interface ready (by Netplan's standards) and so you'll still have that two minute timeout. The only way around the two minute timeout is to either not mention the interface at all in your Netplan configuration or to set it as 'optional: true'. Both of these will cause Netplan to not list the interface as an interface that has to reach the "degraded" state and "configured" status in the systemd-networkd-wait-online configuration file it writes.

(I don't think Netplan ever writes RequiredForOnline=no into a systemd-networkd .network file it creates.)

If you set all interfaces in your Netplan setup to 'optional: true', systemd-networkd-wait-online won't run at all under normal circumstances. This isn't obvious from the created configuration; Netplan will write a drop in configuration file for it that has no ExecStart= lines, which makes it seem like the standard .service file should take over and run systemd-networkd-wait-online in the standard way. However, the drop in file that Netplan writes does have a conditional dependency on /run/systemd/generator/network-online.target.wants/systemd-networkd-wait-online.service being a symbolic link, and Netplan doesn't set up that symbolic link, so the whole service is skipped. Given this, I think you almost certainly shouldn't set 'optional: true' on all interfaces, which means not setting it on your main one (the one you're actually using).

(That systemd-networkd-wait-online doesn't run at all is obvious if you look at the logs, which will report that it's not running because its conditions aren't met.)

Suppose, not hypothetically, that you have some standard physical servers. As is typical, the servers have more than one network interface (two is usually the basics even for 1U servers, and maybe you put in a 10G-T card in some), but you're only using one network interface on one network; the others are just sort of there. Recently you've started putting Ubuntu 26.04 LTS on them, and to your surprise (and displeasure) these machines stall for two minutes on boot, which they report as waiting for systemd-networkd-wait-online.

The direct culprit is that you haven't told Canonical Netplan to ignore the state of network carrier for all of the network interfaces that you aren't using and don't have plugged in to anything. This is the 'ignore-carrier' property, listed in Properties for all device types, and you need to set it on every network interface that is listed in /etc/netplan/*.yaml. However, why this happens and how we got here is a little bit convoluted (and it can happen in versions other than 26.04, under the right circumstances).

Update: My fix is incomplete. See How to correctly not wait for network carrier in Netplan.

Famously, Netplan doesn't actually configure your networks itself. Instead it creates a configuration for something else (a 'backend'), and on servers this is systemd-networkd. For every network listed in your Netplan configuration, Netplan will write a '10-netplan-<name>.network' file to /run/systemd/network/, with the appropriate contents taken from whatever your configuration says. As sort of covered in systemd-networkd-wait-online's documentation (when carefully combined with the description of network states in networkctl), this will cause systemd-networkd-wait-online to wait for every such interface to have carrier (up to its timeout), unless the .network file specifies ConfigureWithoutCarrier=yes, which Netplan sets only if you put the ignore-carrier property on the interface in your Netplan configuration.

(Under some circumstances Netplan will also write .link files for one or more interfaces.)

At install time, the Ubuntu server installer gives you options for what to do with each network interface it detects. If you explicitly configure an IP address or set an interface to do DHCP (or the installer sets it to do DHCP and you don't explicitly turn that off), the interface will of course appear in your installed system's Netplan configuration because it has to. However, if you explicitly disable an interface (or in some versions of the installer, the installer disables it because it detected no carrier and/or no DHCP), the Ubuntu server installer has traditionally not written any information about that interface to your installed server Netplan configuration. Since these additional interfaces weren't mentioned in Netplan at all, Netplan never wrote out .network files for them and systemd-networkd-wait-online never waited for carrier for them. Your server's Netplan file contained only the interfaces you actually used and you were happy even if there were no 'ignore-carrier: true' settings.

What's changed in Ubuntu 26.04 is that the Ubuntu server installer appears to write out an 'accept-ra: false' property for every interface it detects, even if you disabled the interface during network configuration in the installer. Since the installer itself now mentions every interface in your generated Netplan configuration, Netplan writes .network files for all of them to /run/systemd/network and you wind up 'waiting' for carrier on every interface on boot, causing a two minute delay, which is the default timeout used by systemd-networkd-wait-online (it's the --timeout option's default value). You can fix this either by setting 'ignore-carrier: true' on every such interface or by deleting all of those interfaces from your Netplan configuration.

(To confuse you, systemd will report during boot that systemd-networkd-wait-online has no timeout and it's technically correct. What systemd really means is that the systemd-networkd-wait-online.service file doesn't specify a TimeoutStartSec= setting and as 'Type=oneshot service, it doesn't get a default one. Instead the timeout is inside the systemd-networkd-wait-online program and is opaque to systemd (the PID 1 daemon that is trying to 'start' this weird service).)

Because I had to wrangle this, here is how to list every network interface and then set them to ignore carrier, assuming you're using Ubuntu 26.04 where they are already all listed in your Netplan configuration:

ifnames="$(ip --json link show | jq -r '.[] | select(.link_type == "ether") | .ifname')"
for i in $ifnames; do
    netplan set "ethernets.$i.ignore-carrier=true"
done

This isn't perfect and will match some virtual interfaces, but I leave dealing with that as an exercise to the reader, along with excluding interfaces that currently have carrier. For our purposes at install time, this is good enough. And yes, we turn off detecting carrier even on our active interface because we would rather have the machine come up to the extent that maybe we can log in on the console or the like.

(It would be nice to have a way to delete a particular interface entirely, but Netplan's command line interface doesn't provide any good way to do that.)

If you have NFS mounts on a client using a sufficiently recent kernel, such as that shipped with Ubuntu 26.04 LTS, and you inspect /proc/mounts (or 'mount -t nfs'), you'll probably discover a new NFS mount option that is listed for your mounts. For example:

[...]:/h/281 /h/281 nfs4 rw,[...],fatal_neterrors=none,[...]

You might be curious what this option is, especially if you have code that parses NFS (v4) options and sorts options into ones you care about and ones you don't (as we do). Unfortunately, as I write this about a year after the option was added to the Linux kernel, 'fatal_neterrors' isn't documented in any manual page that I can readily find. As far as I know, the only accessible documentation for it is in the commit that added it, which I'm just going to quote:

NFS: Add a mount option to make ENETUNREACH errors fatal

If the NFS client was initially created in a container, and that container is torn down, there is usually no possibility to go back and destroy any NFS clients that are hung because their virtual network devices have been unlinked.

Add a flag that tells the NFS client that in these circumstances, it should treat ENETDOWN and ENETUNREACH errors as fatal to the NFS client.

The option defaults to being on when the mount happens from inside a net namespace that is not "init_net".

As you can see in that commit, currently the only two values you can see for this are 'fatal_neterrors=none' or 'fatal_neterrors=ENETDOWN:ENETUNREACH'. As a mount option that you provide manually, you can also ask for 'default' or 'ENETUNREACH:ENETDOWN'.

If I'm reading the tea leaves correctly, this commit first appeared in Linux 6.15.0 (and so a 'fatal_neterrors' option for NFS mounts first started being reported in /proc/mounts then). This specific commit seems to be merely one of a whole series of net/sunrpc and fs/nfs commits that make ENETDOWN and ENETUNREACH errors as 'fatal' under the right magic circumstances, and I think that by 'fatal' what it means is that any client NFS request fails immediately and isn't ever retried. I don't think it makes the NFS mount itself disappear. Hopefully it will allow you to unmount the now permanently non-functional NFS mount.

(The current state of affairs is that all NFS mounts will report some value for the 'fatal_neterrors' option, although the value of 'none' is effectively the pre-6.15 state of affairs.)

PS: Probably the place where documentation will appear at some point is the upstream source for nfs(5), which will then wind up on man7.org as nfs(5).

We have a long established approach for configuring our Linux systems, and specifically our Ubuntu systems (since these days the only Linux we run is Ubuntu). Part of that has been that we use a standard /etc/resolv.conf that points to our local DNS resolvers (which have our split horizon DNS setup). However, as people may have noticed, Linux distributions these days are moving to systemd-resolved instead of a straight /etc/resolv.conf setup.

There are good reasons for this switch in general. Systemd-resolved can cope with a variety of situations and problems that trip up the standard resolv.conf setup and it cooperates better with other programs. Almost none of these apply in our particular situation for our own servers, which have a static network configuration, a static DNS resolution search domain, and our DNS resolvers already handle all of the split resolution (and we had to make this work well before systemd-resolved existed).

(The one aspect of systemd-resolved that matters to us is that if you have multiple DNS resolvers, as we do, resolved will rapidly fail over from one that isn't responding to one that is. Resolved can also cache answers locally but for us this isn't a feature and we'd turn it off.)

Although I switched to systemd-resolved on my own desktops (cf), our servers have traditionally stuck with our static /etc/resolv.conf setup and entirely turned off systemd-resolved (which means even its D-Bus interface isn't available, because we'd rather have D-Bus based DNS lookups fail entirely than have them return different results). This isn't the Ubuntu default (which has been systemd-resolved for a number of LTS releases). So far we've been able to get Ubuntu to accept this without complaint, but as time goes on I've been feeling more and more nervous about going the /etc/resolv.conf path, especially without a working D-Bus systemd-resolved setup.

The reason I'm nervous is the traditional issue of people writing and testing software only to and against the default system environment. Some day we're going to find some piece of software that simply assumes that it can make D-Bus DNS queries to systemd-resolved; if we're lucky, the software will explicitly state this as a requirement. And some day there will probably be software that relies on some aspect of systemd-resolved's behavior even while doing traditional non-D-Bus name resolution, such as expecting the special '_outbound' name to resolve (although that's not universally available even on systemd-resolved hosts).

It's possible to configure systemd-resolved so that it only handles DNS resolution for people who ask over D-Bus. But if you're going to do that, you need to make sure that the systemd-resolved configuration (for DNS resolvers and DNS search path) matches your /etc/resolv.conf, and once you have a matching configuration, perhaps it's simpler and less error-prone to use systemd-resolved for everything. That gets you (us) on to the theoretical happy path of using the setup that Ubuntu expects (at the cost of having to care about the DNS settings in /etc/netplan files, and possibly having to update them if you ever, say, change or add a DNS resolver).

Through Ubuntu 24.04 LTS, we explicitly masked and disabled systemd-resolved and used our /etc/resolv.conf. In our current experimental Ubuntu 26.04 LTS setup, we first arranged to configure systemd-resolved (via /etc/netplan) to match our resolv.conf so that D-Bus resolution would work if anyone explicitly tried to use it, and now we're trying out having systemd-resolved be the normal resolution method. This makes our 26.04 machines more like Ubuntu expects at the cost of being different from our 24.04 machines, and maybe exposing them to oddities in systemd-resolved's behaviour. We'll have to see how it goes (and we may go back to our /etc/resolv.conf ways).

(The one thing we don't do is have nss-resolve be used in /etc/nsswitch.conf. I don't even know if putting it there is stock Ubuntu 26.04 behavior, and I don't even use it on my desktops so I have no experience with it.)

PS: We're not planning to make any use of systemd-resolved DNS server delegations. All of that DNS steering is going to continue to live only on our DNS resolvers.

(One comment.)

If you look at it from the right angle, the existence of third party package systems that sit on top of Linux distributions is rather odd. After all, these distributions already have packaging systems, yet here people are, ignoring them and writing new ones. A while back on the Fediverse, I said something on this general topic:

I have feelings and some of the feelings are that everyone screwed this up, for actually natural reasons. I don't think any distro solved the 'how do we let people easily build and deploy software on us' problem, and so programmers do what programmers do and built themselves another layer of indirection to solve their immediate problem.

(I mean, ship an entire container running a web server to do what you could do with a CGI or some PHP files or ... this is my face, etc.)

(There are two sides of this, the system side (what I was talking about) and the user side.)

On the one hand this is perfectly natural for Linux distributions to do. They built their package system to manage their own components, with all of the features that are important for that, not to make it easy for programmers to create a package for something that was the contents of '/opt/<whatever>' and specified a few entry points and system dependencies. On the other hand, this is part of how we got Docker and also a bunch of third party package managers for things, because programmers really do want and need something that is that simple, and they're quite willing to write it themselves.

(I'm using Docker as an example because it has a simple system for specifying how to build your software and then declaring various entry points to it. This is a lot of what a package manager does but almost all of them are more complex.)

At the same time, what I said is too harsh on Linux distributions, because the two problems containerization solves are hard, complex ones. Containers provide a hard isolation boundary for changes to the system and avoid the need for a huge API for system things. My aside of 'a CGI or some PHP files' elides a huge level of complication for what a hypothetical programmer built system package of the same application would have to specify, and the API that specification implies. If you can hermetically package and then deploy a web application as some PHP files in /opt/whatever, the system has to have already specified a lot about how that's connected to everything, how installing your package may trigger dependencies of installing a web server and maybe configuring it for TLS and so on.

You could do a simple 'package and deploy' system, but it would be limited in what it could deploy, specifically in how it could connect the raw files it stuck in /opt/whatever to the system (because raw files on their own aren't all that useful). These limits would probably have driven programmers to reinvent many or all of the third party package and container systems that we have today.

(You absolutely would need 'easy and simple', because programmers voted with their feet that system package managers were too complicated to deal with. You've always been able to build your own local software as RPMs or .debs and manage them that way, but almost no one actually did that because it was too much of a pain.)

With that said, I do wonder how far you could get today on Linux if you had people provide 'entry points' in the form of systemd units in some standard place in their '/opt/<whatever>' directory tree tarball. Systemd units give you service activation, service dependencies, the equivalent of cron entries in timer units, and let you expose network services as socket units. You can't create fixed users and groups but you can ask for dynamic ones that are created on the fly, and systemd could probably have something to put all of your units under some UID that's not root and has less powers.

(Systemd has some general features for this such as portable services, but it envisions a container like experience where you build and ship a full OS image, even if it's a small one.)

(Sometimes I find that my off the cuff gut reactions aren't quite as well baked as I thought.)

(3 comments.)

In a comment on my entry on universities, email, and the issues of running things in house, I mentioned that our departmental email system has a non-trivial cost in both hardware and staff time. For the hardware costs I can easily count servers, but things are more fuzzy for the staff time side of things so I'm not going to try to come up with a number (especially because our email system itself is somewhat complex because we have a number of unusual features, such as our simple mailing lists). The bigger issue is that the basic time cost to maintain your email system is in some sense an illusion, or at least an inadequate measure.

The reality of modern email is that to run a modern email system you need to know about a lot of stuff, and worse the stuff that you need to know about keeps evolving. You can't set up a mail system and then walk away from it apart from software and hardware upgrades; instead, you have to keep on top of a perpetually changing and evolving landscape of anti-spam systems and especially what you need to keep your outgoing email being delivered. SPF, DKIM, DMARC, DMARC alignment, and so on are merely today's names; in a few years there will be more, different things that are necessary to know and deal with, which will require you to modify your mail system as part of its obvious time requirements.

Obtaining this knowledge and keeping up on developments in email is often more or less invisible time. Without it, your system may work today (especially if you set it up from a good guide) but things get riskier as time goes on. At best, you notice when outgoing email starts bouncing; at worse, you have no idea until people start complaining to you (if you're lucky) that email they send isn't getting received properly. Then, if you're lucky, you can find another guide for the current new reality of email; otherwise, you're in for research and learning on your own. To copy from an old story, your time spent might be half an hour to make the eventual change to your mail system but a day of research to know what to change and why.

(A similar thing exists with anti-spam and anti-malware handling for incoming email. In a modest sized environment (such as ours) you have no realistic choice but to outsource much of that to some free or paid piece of software, but that still leaves you to monitor the overall situation, determine when your current software is falling behind, and then find and configure new software.)

Of course you need to develop and maintain a certain amount of expertise with your mail software, but that's usually easier to keep current. Your mail software typically makes changes less frequently than the overall Internet mail environment does, and those changes will often be explicitly documented in release notes, news entries, and changelogs. Still, depending on how much complexity you opt for in your mail system, this may take a significant amount of time initially (learning Exim was in no way an overnight thing for me, and then I had to design and build our configuration because Exim is a construction kit).

(We have what I certainly hope is an unusually complicated mail system (also, also).)

If you have a multi-person system administration team and only one person from it goes through all of this and learns all about email (both things like DKIM and things like your local mail setup), what you've got is a single point of expertise. If that person is on vacation and a mail problem comes up, you kind of have a problem. To avoid this, you'll need more staff time for more people to be at least somewhat up to speed on email stuff and how to deal with problems. At one level this is no different than any other system you operate, but those systems probably evolve slower than Internet mail does and require less time to keep up on.

None of this is insurmountable. But it's not trivial either. Maintaining your own mail system (in the broad sense) is going to take a non-trivial amount of staff time for one or more people to keep up on Internet mail developments, monitor your mail system for signs of problems like too much incoming spam and malware getting through or too little outgoing mail reaching its destinations, and troubleshoot issues. This need for maintained expertise is part of why it can be simpler to outsource email to specialists who deal with it a lot.

(Also, as you deal with a larger population of people using your systems, you also need to worry about compromised accounts sending out bad stuff.)

(One comment.)

I've been using my home desktop for many years now, with a variety of disk drives, and over those years some of those disks have failed (without data loss, because I use software RAID mirrors). Some of the time, those failures have happened when the HDD was still under warranty, but despite that I've never bothered to try to return one for a warranty replacement. This is strikingly unlike work, where we routinely return failed drives that are still under warranty.

Historically there have been two things that changed my behavior at home. The first is that when a disk drive fails in my home system, my priority has always been to restore redundancy as fast as possible. Sending a drive in for warranty replacement and waiting for the new one would take weeks, and I don't want to run a non-redundant setup for that long, so I've always gone out and bought a replacement drive immediately. The second is that a lot of the time, the failed disk was old enough that the replacement HDD I'd be buying was bigger, which typically meant that I had little immediate use for a smaller than I now wanted warranty replacement drive.

(In retrospect, spare HDDs for backups were one use, but I'm lazy.)

Neither of those factors have historically applied at work. At work, we've always maintained a spares pool for disk drives, so we already have an immediate replacement for a failed drive and then the new warranty replacement drive can be cycled back in to the spares pool. As for the potential size mismatch, there are two mitigating factors; we haven't changed the size of our disk drives very much, and we can usually find a use for smaller drives as system disks and the like. System disks for our servers merely need to be 'big enough', and it's okay if they're mismatched in size (although we prefer them to be matched).

This generalizes for more than disk drives, but disk drives are the thing I've had fail at home so far. If I had a power supply failure or a CPU failure or the like in my home desktop, I wouldn't be able to wait for a warranty replacement; I'd be buying something right away.

(These days that argues strongly for keeping older, good enough hardware around to use as a fallback, as new hardware becomes increasingly unaffordable. If I had a spare desktop I could press into service at home, I could wait out a warranty replacement. Although my current home desktop is so old that any five year warranties expired years ago.)

At work, we make sure we have spares for servers and other important things even though it costs more, because we have to. One reason we can afford to do this is that we don't need all that many spares; a couple of disk drives or servers can be the spares for a whole collection of in production ones, on the assumption that we won't lose too many at once. At home, I have so few disk drives, desktops, and so on that a 'spares pool' would be much bigger relative to my active stuff.

(This is partly because I have only a single home desktop. There's lot of people with multiple home machines and so on.)

PS: Long warranties are still a useful signal from vendors even if I may not take advantage of them at home. A disk drive with only a three year warranty instead of a five year one may well last much longer than five years, but that difference is a suggestive message.

(2 comments.)

Back in 2022 I said that my Firefox addons hadn't changed in a while, and while that's still mostly true there's been some minor changes that I want to write down. Addons are central to my Firefox experience, but I don't change them very often partly because of that and partly because I'm much more conservative with Firefox than I am with things like Emacs packages. I'm generally happy with my Firefox environment and most of what I do with it is stop irritating changes and fix things that get turned off.

Of my old addons in my main Firefox setup, I still use Foxy Gestures, uBlock Origin, uMatrix (which is still not quite dead), Cookie AutoDelete, Stylus, Textern, Certainly Something, HTTP/2 Indicator, ClearURLs (although that may be ineffective these days), and Open in Browser, although I'm not sure that's doing anything for me. I no longer use HTTPS Everywhere. Because I switched from Firefox's default "storage partitioning" for cookies to the stricter "first party isolation", I had to switch from Cookie Quick Manager, which I don't think works with either of them, to Cookie Manager, which does work with first party isolation as far as I can tell. In my main Firefox, this is the only addon change, although in a sense there's another change in that I'm now pervasively using Stylus.

I have a second significant Firefox environment and in this environment I recently set up a container-based setup. This setup uses Multi-Account Containers for the basic multi-container support and FoxyTab to assign wildcarded domains to specific containers. This second Firefox now also has Cookie AutoDelete (it didn't used to), and also I've installed a few other addons over time: Copy Alt Text, Don't Fuck With Paste, Right-Click Borescope, and User-Agent Switcher and Manager. I have all of these in this Firefox because this Firefox allows JavaScript generally, which allows websites to do a lot more things that I want to override and examine. My normal Firefox blocks JavaScript, so websites can't readily do things like blocking paste. I don't use these addons very much, so they're mostly around in case I need them.

Although it's not a change in addons, a significant change in my Firefox usage is that my second Firefox environment has moved from being disposable (with everything discarded when Firefox shuts down) to being persistent, because I admitted to myself that that's what I was doing with it. This is part of what pushed me to add Cookie AutoDelete to this Firefox profile; even before I made it persistent, I admitted that I wasn't shutting it down very often and so cookies were increasingly lingering in it.

(4 comments.)

So today I had a learning experience:

Firefox Nightly apparently has decided to not allow extensions/addons to run for file:/// URLs and I am sorry, this completely breaks my environment (my startup page is a file:/// URL) and I find it infuriating. Maybe I can find some hidden toggle for this somewhere.

This is relevant to me because my start page is a local HTML file and I'm a long time user of Foxy Gestures (cf), which I expect to work on my home page. When it doesn't, I get upset.

This turns out to be bug 2034168, Add explicit permission for file:-access and implement extension.isAllowedFileSchemeAccess(), the components of which landed in Firefox Nightly on May 29th. Fortunately there's a way to allow extensions to still work on file: pages, but it's a bit annoying. Extensions can have per-extension additional permissions, and this change adds a new such permission, "Access local files on your computer". But since this is a per-extension setting, you have to go through each of your extensions (at least the ones this is relevant for) and turn it on, and this permission landed only with the change, so the first time you start Firefox all your extensions will be broken.

(One hopes you don't count on something like uBlock Origin to block JavaScript embedded in local HTML pages from doing undesired stuff. Well, and have them open in your Firefox so they'll be immediately (re)loaded when Firefox starts.)

I can't really blame the Firefox developers for this. The new situation is probably more secure and likely very few people will actually notice (and it's going to be in the release notes for Firefox 153, for those of you who read them). But this is one reason why I have a whole test procedure for my custom Firefox trunk builds before I actually use them for my regular browsing.

(At this point I will give a shout out to mozregression, which allowed me to easily find the exact point where this changed and thus find bug 2034168. If you try out Firefox Nightly on a regular basis, you might want to keep it around, perhaps via pipx.)

On the positive side, this may lead to Firefox addressing bug 1617594, Allow extensions to create tabs with file:/// URLs with an explicit permission, which would reduce the need for one of my custom Firefox changes. Although I already have that change and would probably keep carrying it in my custom Firefox builds, since I have other tweaks as well (cf).

PS: Firefox's motivation here seems to be not that local file-based content is untrusted but instead that it's highly trusted, and because it's highly trusted it's potentially dangerous to let extensions change what people see. Firefox has a similar but stronger block on extensions running on Firefox websites like addons.mozilla.org.

When I wrote about the Emacs packages I use, I mentioned that I had Embark installed but barely used it because I didn't understand much about how to really use it. One reason for that is that while there are a bunch of articles on the web about things you can do with Embark, all of the ones I've tried to read started out with complicated stuff involving other third party packages I didn't use, which caused me to tune out and stop reading. As sometimes happens, writing that entry caused me to poke at Embark some more and now I have a somewhat better understanding of it and some Embark tricks I want to remember.

(Now that I've made the effort to read it, Fifteen ways to use Embark has a bunch of useful examples that use only built in packages. Also, part of my confusion is that Embark actually does multiple things.)

With Embark, you start with a 'something' (what Embark calls a target) and then Embark lets you to do an assortment of things to it; some Embark writeups describe this as a middle mouse button context menu. There are at least two ways to get the 'something' (cf). In a regular buffer, it's whatever is at point (or the region if you have one active); in the minibuffer, it's whatever you're entering or completing. To add a bit of confusion for regular buffers, you can often change what Embark is acting on. For instance, if the GNU Emacs point (cursor) is on a word, Embark can act on the 'word' (in various ways depending on what it thinks the word is), the sentence it's part of, or the paragraph. What Embark can do depends on what sort of thing it has as its current target, so it offers you a completely different set of options for the name of an ELisp function than for a file name (see Default Actions for a very large list).

(When Embark starts in a situation where there are multiple options for the target, it will talk about 'shadowed targets at point' in the Embark buffer.)

One important 'action' that I want to remember that's always available is M-x, ie run a suitable (Lisp) command, and in fact a lot of your regular keybindings will apparently work. Not all commands will work right when run by Embark, but most of the ones you probably want to run will; see How does Embark call the actions? for the gory details.

To use Embark, you need a target. One way to get a target if you don't already have one at hand is to type it into a buffer, but another, better way is to use the minibuffer, by triggering some GNU Emacs command that will prompt you for the type of thing you're interested in. This leads to some of my 'stupid' Embark tricks (which are apparently perfectly normal). For example, suppose that I want to toggle the value of a GNU Emacs setting variable. Embark provides this as an action on variables, so the quick way to do this is 'C-h v', which will start minibuffer completion for variable names, then when I've picked the variable, trigger Embark and pick 't'. Similarly, you can set the value of a variable via Embark rather than having to remember 'M-x set-variable' and then completing the variable name anyway.

A bunch of Embark documentation talks about using Embark in the minibuffer because you changed your mind about what you want to do. You start out doing C-x C-f to open a new file and then you realize you want it in a new Emacs window so you can see your current file and the new one at the same time, so you use Embark to switch the result of C-x C-f to 'open file in new window'. Some specific options you can switch to are explicitly available, but in general you can switch to anything, although it's on you to make sure that your minibuffer completion makes sense for what you're switching to.

The logical extension of this is to not bother using or maybe even remembering C-x 4 C-f for 'open file in new window' and always using C-x C-f and then Embark to get it. Much as with my 'toggle a variable' example, you (I) are using C-x C-f as a way to generate file names for Embark to act on. Anything that generates file names in the minibuffer would do, but C-x C-f is a harmless thing if you hit RET by accident instead of triggering Embark. This is an intended use of Embark, per this Fediverse post, which I'll quote a bit of:

[...] The pattern is that any command that prompts you for Xs becomes an X manager. [...]

Functions, variables, files and directories, buffers, GNU Emacs packages, and so on, you can trigger something that prompts you for one of them, use all your completion features to fill it in, and then use Embark. You could even build a collection of personal commands (and keybindings) that only prompted you for the appropriate thing and then did nothing with the result.

A related trick is that you can use minibuffer completion to complete things you're writing in regular buffers, through Embark's action to insert text from the minibuffer into the regular buffer. Do you want to insert a file name into what you're writing? Use C-x C-f to trigger filename completion in the minibuffer and then Embark's general 'i' action to insert the result in your text. If you already have a completion setup with good completion for regular buffers (as I do), this is most useful for types of completion that aren't offered for your current buffer. In text buffers, this will be most of them; in code buffers this is likely to be things like file names.

(For file names specifically you can get the same completion option with Cape, although using your completion at point setup instead of minibuffer completion. But the Embark trick works for absolutely anything you can trigger a minibuffer completion for, including custom things.)

Another trick is that in minibuffer completion, Embark can also act on the current completion candidates, applying some action to all of them instead of just to one of them, the way it would if you finished completion. There's a number of actions Embark provides for acting on these groups, including exporting the current set of candidates to a buffer where you can further manipulate them in various ways that depend on the types of things (and whether you do an 'export' or a 'collect'). Embark also lets you create ad-hoc collections of things for it to act on. I'm writing about this because I looked it up but I don't think I'm likely to use this particular aspect of Embark very much, because it seems pretty fiddly.

(VOMPECCC: A Modular Completion Framework for Emacs has a discussion of the advantages of these Embark collection buffers.)

Unfortunately, using Embark in text mode buffers is somewhat fiddly because Embark often has unusual ideas of what a text word actually is. If you're lucky, Embark decides that it's an identifier and offers you various useful options (and also highlight other occurrences of the word). If you're not lucky, Embark will decide that your word is some other type of thing with a restricted set of actions; for example, 'minibuffer' (as a bare word) will be taken as an Emacs Lisp library, which has only a restricted list of actions. As far as I know there's built in way to change the type of thing or add an option to act on it as another type.

Since this is GNU Emacs, we can use violence, which is to say we can define a new sort of target, call it a 'word', and add a keymap for it that has specific bindings we want. This requires following the examples of both adding a new target and defining a keymap:

 (defvar-keymap embark-word-map
   :doc "Keymap for Embark actions on plain words."
   :parent embark-general-map
   ;; TODO: What should RET do?
   "o" 'occur
   "$" 'ispell-word
   "'" 'expand-abbrev
   "p" 'embark-previous-symbol
   "n" 'embark-next-symbol
   "c" 'capitalize-word
   "l" 'downcase-word
   "u" 'upcase-word
   "H" 'embark-toggle-highlight)
 (add-to-list 'embark-keymap-alist '(word . embark-word-map))

 (defun embark-target-word-at-point ()
   "Target a word at point but only in text mode buffers."
   (save-excursion
     (let* ((start (progn (skip-chars-backward "[:alnum:]") (point)))
          (end (progn (skip-chars-forward "[:alnum:]") (point)))
          (str (buffer-substring-no-properties start end)))
       (when (and (not (string-empty-p str))
                  (eq major-mode 'text-mode))
         `(word ,str ,start . ,end)))))
 (add-to-list 'embark-target-finders 'embark-target-word-at-point)

(In an ideal world this might also look to see if it was in comments or strings in a prog-mode buffer, but that's too much work for this quick hack.)

My 'word' target isn't quite as deluxe an experience as you get with identifiers, because identifiers and symbols will also lazily highlight all other occurrences in the buffer. But possibly you don't want that for plain words.

(Embark does say that it's primarily for minibuffer stuff, it's right in the name: "Emacs Mini-Buffer Actions Rooted in Keymaps".)

Sidebar: Giving Embark a connection to Flycheck

Embark ships with a connection to Flymake, so you can trigger Embark with point on a Flymake diagnostic and get some useful bindings. Because I default to Flycheck, I wired up the same thing for Flycheck, and to save other people having to do the work, here it is:

 (embark-define-overlay-target flycheck flycheck-overlay)
 (defvar-keymap embark-flycheck-map
   :doc "Keymap for Embark actions on Flycheck diagnostics."
   :parent embark-general-map
   "RET" 'flycheck-list-errors
   "e" 'flycheck-explain-error-at-point
   "h" 'flycheck-display-error-at-point
   "n" 'flycheck-next-error
   "p" 'flycheck-previous-error)

 (add-to-list 'embark-target-finders 'embark-target-flycheck-at-point)
 (add-to-list 'embark-keymap-alist '(flycheck . embark-flycheck-map))

Add more Flycheck bindings to taste, those seemed to be the obvious ones to me.

One piece of technology news of the time interval is that Google is no longer going to be providing Internet search, it's going to be providing 'answers'. On the Fediverse, I had a reaction to that:

Given Google's apparent change to what Google Search will be, I'm wondering if I should even allow Googlebot to crawl my techblog. If they're going to AI slop up anything they tell people after a search, I'd rather have my writing excluded entirely rather than be garbled.

(I have no idea how much traffic I get from people using Google Search, and if Google is going to do this I feel like I should punish them by pushing people away from them, even a little bit.)

To echo what Paul Cantrell said on the Fediverse, we tolerated and even embraced Googlebot crawling our websites as part of a social bargain. Allowing Googlebot was a big part of how people found our sites and our work, which is to say through searches. That social bargain has been fading as Google put more and more things inline, but at least they were still providing links and directly showing our words (I know, usually).

But now Google has said out loud that Googlebot is just the front end crawling ingester to an LLM system, much like all of the other LLM crawlers that are hammering our sites. Google is no longer in the search business, where they provide links to people; they're in the 'answers' business (which is to say, the probabilistic text generation business). Much as with HTTP requests from cloud provider IPs, this raises the question of whether we should care about allowing Googlebot to access our websites or whether it's now a source of undesired crawl load, or at least of no meaningful benefit to our sites.

(A hasty clarification: by 'we' I mean people running small web servers and web sites, as with requests from cloud provider IPs.)

It feels practically heretical to say this; as recently as last November I was saying that people needed Google to crawl them. However, here we are. It no longer feels at all obvious that I'm going to get future benefits from allowing Google to crawl Wandering Thoughts. For that matter I don't know if I'm getting current benefits (well, if people are in general), since you can't really tell any more when people come to your site from Google Search.

(If you can still tell from the Referer HTTP header, that suggests that almost no one is coming here from Google Search. Googlebot is still (re)crawling portions of Wandering Thoughts on an ongoing basis; in fact, now that I look it regularly fetches the front page, although it politely uses HTTP conditional GET.)

(5 comments.)

One of the problems I have when talking about 'vim' is that in practice, there are several versions of 'vim'. I don't mean 'version' in the sense of release numbers (although Vim has lots of releases and things change in them over time), but more in how people approach and use Vim. Even more than GNU Emacs, I feel that Vim has turned into an editor that has multiple faces, so the 'vim' that you use may not be the 'vim' that I use.

(Which version or face of Vim you see depends in part on what your system decided to set as Vim defaults, unless you're advanced enough in Vim things to have a personal configuration that overrides them all.)

I've seen people talk about at least three broad versions of Vim:

• The Vim with a bunch of plugins that is a full blown code editor or even IDE environment with LSP support (eg, also) and everything else you'd expect from such a thing. My impression is that this is surprisingly popular version of Vim, and I can see why; if you like the Vim editing style and find that LSP stuff fits into it for you, why not?

• The Vim with syntax highlighting, smart indentation (including language dependent things), and other features that you'd expect of a competent programming editor (one without IDE style integration and features).

• The Vim that's only slightly different from true original Vi, 'merely' fixing the things that make Vi a product of its time, such as only having a single level of undo. This is the Vim that I use.

  (People disagree about what in the original Vi needs fixing, and also how, cf undo.)

The Vim I use doesn't have syntax highlighting or smart indentation and I wouldn't use it if it did (or at least if those were on by default). But the Vim that other people experience does sometimes have those things, not necessarily by their decision but because that's how Vim comes set up on that system. Someday, someone may ship a system where Vim is pre-configured to provide the full IDE experience with LSP support already added and so on.

One of the issues with this is that because Vim can be configured to be quite different things, it's not necessarily obvious to people (me included, cf, also) how to change the Vim they get by default on a system to a Vim they want. People new to Vim may not even know it's possible, and people who know it's vaguely possible may conclude that it's too much work and they might as well switch editors, either to another version of Vi or to an entirely different editor.

At work, we have a lot of basic servers. These basic servers are 1U rack servers, with unexciting amounts of disk bays, RAM, and CPU performance, because we buy them to do straightforward jobs (for example, much of our mail system hardware is basic servers). Right now, these are all x86 machines, but like a lot of people I'd love it if the x86 architecture had real competition (even if returning to the days of a multi-architecture Unix environment wouldn't necessarily be fun). However, these days I don't think we're likely to see basic 1U ARM servers that we're interested in, and why that is comes down to modern hardware and modern basic server power usage.

At this point, the largest hardware difference between a reasonably performing basic x86 server and a reasonably performing basic ARM server will be the CPU (and surrounding chipset). Everyone uses the same RAM, the same PCIe busses, and quite possibly the same supporting chips for things like Ethernet and so on. And everyone is going into the same rack form factor 1U boxes with very similar power supplies and physical sizes.

My impression is that at the basic end, a comparably performing ARM server CPU is going to cost about the same as an x86 CPU (assuming you can get the former). My further impression is that the ARM CPU isn't going to have a significant power and heat advantage under normal usage, especially since modern basic servers seem to have pretty low power consumption in general (also). This means that I'd expect a basic ARM server to cost about the same as a basic x86 server and generate about the same heat and power load. Even if the ARM CPU is a bit cheaper, it's only one component out of many (and let's ignore the current inflation of RAM prices).

More powerful ARM servers (such as those used by the cloud vendors, or the ones with powerful CPUs that you can theoretically buy) can have a real advantage in some situations over x86 servers, so if you use some of those you might want some basic ARM servers for lighter weight work on the same architecture. But if your environment is otherwise free of ARM servers, and the basic ones cost the same as x86 servers, and maybe you have a bunch of x86 servers already, there doesn't seem to be much to attract people to ARM servers.

I feel a bit sad about this, because there's part of me that thinks it would be nifty to have some ARM servers around, but it certainly makes life easier to only have one CPU architecture for in our Unix machines. x86 CPUs are perfectly good, and these days all of the other parts are basically the same.

(Given this, I'd be somewhat surprised if very many vendors even offered basic ARM servers, especially for a competitive price. There are people selling 1U ARM servers but they're high core count things and I assume they cost accordingly. I don't know if anyone is making ARM servers in the 16 to 32 core range (or less), although there are definitely ARM CPUs that are that small. Although even the larger CPUs can have issues, although apparently that's on an older, lower performing ARM CPU.)

PS: In the past I've read that x86 CPUs no longer really have a power penalty because of their architecture; my impression is that this is less because dealing with the weird x86 stuff got more efficient and more that these days it's dwarfed by everything else going on in your typical complex, out of order cores.

(One comment.)

My Emacs configuration seems to have more or less settled down again after a flurry of changes, so it's time to update my previous list of Emacs packages that I use, so that I can come back to this entry later and see how things have changed over time. A bunch of things haven't changed since last time so I'm going to put the unchanged stuff at the bottom.

Currently I'm using Emacs 30.2 everywhere so some of the things I'm mentioning here are now built in, which is why I'm not restricting this to third party packages. As before I'm going to exclude dependencies that are automatically installed by the Emacs package system (since I don't use them, I just have them around in the background).

In no particular order:

• In text-mode, I use Flyspell (although there's a gotcha there). I've turned off Flyspell's default binding to M-TAB, because I find Flyspell basically useless for actually correcting words. Almost all of what I use Flyspell for is simply marking words it thinks are misspelled or mistakes so I can spot them (and then fix them by hand).

• I now use Eglot as my LSP server environment, which is built in to GNU Emacs. I use LSP servers for Go, Python, shell programming, and to some degree with C (and technically I have YAML and Rust stuff installed, but I don't really work with either in GNU Emacs).

  (I'm not sure a LSP server is all that useful for shell programming, or at least the LSP server I have.)

• For programming diagnostics, I primarily use Flycheck but I also have a Flymake configuration. My Flycheck configuration has flycheck-eglot for integration with Eglot, and also flycheck-golangci-lint (for Go) and flycheck-mypy (for Python) as supplemental linters that I turn on when I want them. For Flymake, I've installed flymake-shellcheck and flymake-flycheck (for when I want to throw the kitchen sink at something in Flymake).

• For Go programming I use go-mode. These days I no longer use supplemental packages for diagnostics; instead, all of that is handled through my fly* package setup, which primarily means the diagnostics provided by the gopls LSP server.

• For completion, I use a whole collection of packages with their own sets of configurations. The current list of main packages is the more or less standard-ish stack of vertico, marginalia, orderless, consult, and corfu, plus completion-preview-mode.

• After a significant amount of wrangling, I managed to get Tramp set up for my unusual environment (see my notes). I'm using it for more things than I entirely expected.

I have three partially used packages for displaying diagnostics and other things in code buffers to make them more visible:

• flyover is a package I just discovered today as my best Flycheck substitute for Flymake's 'flymake-show-diagnostics-at-end-of-line' setting, which I like as an option for shoving diagnostics in my face when I want to be sure I see them all.

• I've installed sideline and supporting packages sideline-flycheck, sideline-eglot, and sideline-flymake in case I some day decide I want a noisy programming mode environment (Eglot or otherwise) that shows LSP code actions, diagnostics and so on all over the place, as lsp-ui could be set to do.

  (I'd like some easy way of filtering Eglot's LSP code actions in sideline because otherwise it's too noisy most of the time.)

• I also have flycheck-inline installed because it's a less obtrusive version of what sideline gives me in Eglot buffers (where the full sideline experience would show me all code actions as well).

Packages that I've carried over unchanged from the previous late 2023 edition are:

• embark, which is in theory a great way to do all sorts of things with a few keystrokes and in practice I mostly use as a handy way to do 'reflow this region' when writing email. I have embark-consult installed as well.

  (I feel as if I should learn more about embark and how to use it well, but there's always so much to learn and remember about GNU Emacs and my configuration.)

• Magit for creating basically all of my Git commits. I mostly don't use Magit for other Git operations, but I consider it essential for easy and flexible Git commits (for example, selective commits). I'll sometimes start Emacs purely to make Git commits with Magit.
• git-timemachine to let me step through historical versions of Git-controlled files in Emacs.

• diminish to turn down the noise level of Emacs' modeline. I configure and use it through use-package so I usually don't think about it.

• backward-forward for easy, web-browser like jumping backward to where I was when I follow a reference to something in lsp-mode. I wrote an entry about jumping backward and forward.

• which-key, which gives me a prompt of what my next options are in multi-key sequences; I find this very useful for things I don't use regularly enough to have memorized or wired into my fingers already. Which-key is now built in to GNU Emacs.

• vundo to give me an easy way to navigate backward through Emacs' sometimes unpredictable undo stack. I know that there are more elaborate packages, like undo-tree, but vundo is quite simple and meets my desires.

• smartparens to make it less error prone to write and edit Lisp (I no longer have it enabled for Python because I found it too irritating). Smartparens isn't perfect for Lisp, but it's broadly better than trying to do it by hand. I don't use any key bindings for it or any of its smart commands (or its strict mode), I just let it automatically insert closing things for me. Some of its rearrangement commands might make my life easier, but life is full of Emacs things to learn.

  (One area of Lisp where smartparens falls down is single quotes, which in my Lisp are most often not paired but instead used to quote symbols. So every time I write "'thing" in Emacs Lisp I have to remove the trailing quote afterward. I'll live with it, though.)

• try, a handy way to try out an Emacs package without going through the effort to add it and then remove it again.

Things I'm not really using but still have installed (both carried over from last time):

• expand-region is a little package to expand the Emacs region out to cover increasingly big things. I use it partly for exactly that, but also partly as a way of seeing where, for example, Emacs considers the current Lisp s-expression or defun to end; if I expand the region to the entire s-expression, I can just look. I have this bound to C-=, which maybe I'll remember this time around.

  (In theory this is useful, in practice I keep forgetting I have it.)

• fold-this seemed potentially useful and I put together some bindings for it, but in practice I don't seem to touch it. I was planning to use it in conjunction with expand-region (as a quick way of selecting a region to fold).

  Folding feels like something that might be useful for navigating files or seeing an overview of their structure if I can figure out how to use it. But I'm not currently convinced it's the best option for this for me, instead of things like consult-imenu (although I'm not using that either).

I used to have evil installed but I removed it because I wasn't using it at all, partly due to its clash with my Emacs reflexes.

Some of these packages are probably out of date or not ideal, since I set a number of them up some time ago.

(Most of these packages are installed from MELPA, which means I'm generally getting frequent updates on the ones under active development and more or less the latest development version. So far this hasn't been a problem.)

In GNU Emacs, completion of things is a complex subject. There are at least two sorts of completion (in the minibuffer and in buffers you're editing) and many options for how things work. There's a whole ecology of third party packages for changing how both sorts of completion operate, some of which have become built in to GNU Emacs over time. For various reasons (cf) I'm going to write down my current setup for this.

For minibuffer completion I use:

• vertico to show more completion candidates, with some significant customizations and custom code to complete MH folder names, and also some forced completion categories.

• marginalia to add more information about completion candidates, except that I've then arranged for completion of GNU Emacs functions and variables to show less information because I mostly want to see their docstrings, not type and argument information.

  (Doing this isn't a standard marginalia option and instead requires somewhat annoying hacks.)

• orderless to provide more powerful completion options, except I use orderless in an unusually minimal configuration. Orderless technically can be used for completion in buffers, but I don't have that connected up in any useful way at the moment.

• consult to provide more context on various sorts of minibuffer completion results. Consult needs or benefits from having some additional packages wired up, like consult-flycheck, consult-flyspell, consult-eglot, and consult-eglot-embark, although I don't necessarily remember to use them very often.

For as I type in buffer completion, I use:

• completion-preview-mode gives me a shell style completion environment, where I can hit TAB at any time to complete the current prefix (mostly) and I can see what that prefix is as a little ahead-of-cursor annotation. It's really great and is almost always all I really want.

• corfu for 'completion at point' and to some degree as I type autocompletion, across both non-LSP and LSP modes (which is a change from before). I have a relatively restricted Corfu configuration which deliberately dials down how in my face it is. I turn on Corfu as you type autocompletion in prog-mode buffers but not in text mode buffers.

A lot of my completion configuration has been stable for a while. The big recent changes were switching to only using Corfu instead of a mix of Company and Corfu, and my discovery of completion-preview-mode.

This is a lot of packages to customize completion, but that's the modern GNU Emacs way; you have relatively narrowly focused packages that deal with one aspect of a large GNU Emacs feature. If you customize multiple aspects, you wind up with a lot of packages (both primary packages like consult and also secondary packages like consult-eglot that extend the primary package and connect it to other things).

An extremely common pattern for our Ubuntu servers is that they have exactly two disks and we want those two disks to be set up as mirrored system disks, with a UEFI boot partition in each and the root filesystem in a mirrored RAID array taking up all of the rest of the space (and for /boot/efi to come from the first disk). If a system has a single disk, we want the single-disk, non-mirrored version of this; if the system has more than one or two disks, the installer shouldn't try to do anything because we have an unusual system that needs hand holding; it should either stop to ask for help or abort.

As far as I can tell, this isn't something you can readily express in Ubuntu's server installer or really anything else that Canonical offers, although the server installer is entirely capable of creating this layout if you set it up by hand. I'm not sure it's very available in any Linux server installer, although I haven't looked outside of Ubuntu.

(Some installers can run scripts that can rewrite the installer instructions, so you could in theory write a script that sniffed around the system, detected everything, possibly did some initial setup, and then rewrote the installer instructions to exactly have what you wanted. This is what I will politely call a little bit too intricate for us to want to try to build our own trustworthy set of scripts to do this.)

Why I care about this is that by hand disk partitioning is probably the biggest time consumer when installing machines using our customized server ISO (and it's tediously boring). It's also probably the most critical thing that stops us from having a fully hands-off network booting installer. It would be nice if installers would someday do better with built in, fully supported and tested code (as opposed to a dangerous script we have to write ourselves and that's hard to thoroughly test).

Unfortunately I don't expect that to happen any time soon. My strong impression is that Canonical's focus is on cloud installation, where people don't use mirrored system disks and typically have very simple and uniform disk names, so you can predict (for example) that your entire fleet will have its system disk on /dev/sda or on disks from a particular maker. (Or perhaps you automatically generate a per-system installer configuration using knowledge of that system's hardware that's pulled from your inventory tracking system.)

PS: I'm relatively confident that I could write a suitable dangerous script to rewrite the Canonical server installer configuration file to do this under the right circumstances. I'm also confident that my co-workers would rightfully reject us using that script, because the various dangers aren't worth it at our scale. Always remember, cleverness is a trap.

Sidebar: Our assorted disk naming

Our systems are increasingly split between systems that have SATA SSDs and systems with NVMe SSDs (and then I test on virtual machines with virtio 'vdX' disks), which means that some systems will have 'sda' and 'sdb' and some systems will have 'nvme0n1' and 'nvme1n1'. An extra complication for all systems is that if we're installing from a USB stick (instead of, say, network booting), the USB stick will appear as an unpredictable /dev/sdX name. So even with all sdX disks, it's not actually true that 'sda' and 'sdb' are the system disks; one of them might be the USB stick.

(7 comments.)