An interesting little change recently landed in the development version of Go, and so will likely appear in Go 1.26 when it's released. The change is that the builtin new() function will be able to take an expression, not just a type. This change stems from the proposal in issue 45624, which dates back to 2021 (and earlier for earlier proposals). The new specifications language is covered in, for example, this comment on the issue. An example is in the current development documentation for the release notes, but it may not sound very compelling.

A variety of uses came up in the issue discussion, some of which were a surprise to me. One case that's apparently surprisingly common is to start with a pointer and want to make another pointer to a (shallow) copy of its value. With the change to 'new()', this is:

np = new(*p)

Today you can write this as a generic function (apparently often called 'ref()'), or do it with a temporary variable, but in Go 1.26 this will (probably) be a built in feature, and perhaps the Go compiler will be able to optimize it in various ways. This sort of thing is apparently more common than you might expect.

Another obvious use for the new capability is if you're computing a new value and then creating a pointer to it. Right now, this has to be written using a temporary variable:

t := <some expression>
p := &t

With 'new(expr)' this can be written as one line, without a temporary variable (although as before a 'ref()' generic function can do this today).

The usage example from the current documentation is a little bit peculiar, at least as far as providing a motivation for this change. In a slightly modified form, the example is:

type Person struct {
    Name string `json:"name"`
    Age  *int   `json:"age"` // age if known; nil otherwise
}

func newPerson(name string, age int) *Person {
    return &Person{
        Name: name,
	   Age:  new(age),
     }
}

The reason this is a bit peculiar is that today you can write 'Age: &age' and it works the same way. Well, at a semantic level it works the same way. The theoretical but perhaps not practical complication is inlining combined with escape analysis. If newPerson() is inlined into a caller, then the caller's variable for the 'age' parameter may be unused after the (inlined) call to newPerson, and so could get mapped to 'Age: &callervar', which in turn could force escape analysis to put that variable in the heap, which might be less efficient than keeping the variable in the stack (or registers) until right at the end.

A broad language reason is that allowing new() to take an expression removes the special privilege that structs and certain other compound data structures have had, where you could construct pointers to initialized versions of them. Consider:

type ints struct { i int }

[...]
t := 10
ip := &t
isp := &ints{i: 10}

You can create a pointer to the int wrapped in a struct on a single line with no temporary variable, but a pointer to a plain int requires you to materialize a temporary variable. This is a bit annoying.

A pragmatic part of adding this is that people appear to write and use equivalents of new(value) a fair bit. The popularity of an expression is not necessarily the best reason to add a built-in equivalent to the language, but it does suggest that this feature will get used (or will eventually get used, since the existing uses won't exactly get converted instantly for all sorts of reasons).

This strikes me as a perfectly fine change for Go to make. The one thing that's a little bit non-ideal is that 'new()' of constant numbers has less type flexibility than the constant numbers themselves. Consider:

var ui uint
var uip *uint

ui = 10       // okay
uip = new(10) // type mismatch error

The current error that the compiler reports is 'cannot use new(10) (value of type *int) as *uint value in assignment', which is at least relatively straightforward.

(You fix it by casting ('converting') the untyped constant number to whatever you need. The now more relevant than before 'default type' of a constant is covered in the specification section on Constants.)

After yesterday's entry where I discovered it, I went and implemented Tony Finch's exponential rate limiting for HTTP request rate limiting in DWiki, the engine underlying this blog, replacing the more brute force and limited version I had initially implemented. I chose exponential rate limiting over GCRA or leaky buckets because I found it much easier to understand how to set the limits (partly because I'm somewhat familiar with the whole thing from Exim). Exponential rate limiting needed me to pick a period of time and a number of (theoretical) requests that can be made in that time interval, which was easy enough; GCRA 'rate' and 'burst' numbers were less clear to me. However, exponential rate limiting has some slightly surprising things that I want to remember.

(Exponential ratelimits don't have a 'burst' rate as such but you can sort of achieve this by your choice of time intervals.)

In my original simple rate limiting, any rate limit record that had a time outside of my interval was irrelevant and could be dropped in order to reduce space usage (my current approach uses basically the same hack as my syndication feed ratelimits, so I definitely don't want to let its space use grow without bound). This is no longer necessarily true in exponential rate limiting, depending on how big of a rate the record (the source) had built up before it took a break. This old rate 'decays' at a rate I will helpfully put in a table for my own use:

(This is, eg, 'exp(-1)' for we only last saw the source 'interval' time ago.)

Where this becomes especially relevant is if you opt for 'strict' rate limiting instead of 'leaky', where every time the source makes a request you increase its recorded rate even if you reject the request for being rate limited. A high-speed source that insists on hammering you for a while can build up a very large current rate under a strict rate limit policy, and that means its old past behavior can affect it (ie, possibly cause it to be rate limited) well beyond your nominal rate limit interval. Especially with 'strict' rate limiting, you could opt to cap the maximum age a valid record could have and drop everything that you last saw over, say, 3x your interval ago; this would be generous to very high rate old sources, but not too generous (since their old rate would be reduced to 0.05 or less of what it was even if you counted it).

As far as I can see, the behavior with leaky rate limiting and a cost of 1 (for the simple case of all HTTP requests having the same cost) is that if the client keeps pounding away at you, one of its requests will get through on a semi-regular basis. The client will make a successful request, the request will push its rate just over your limit, it will get rate limited some number of times, then enough time will have passed since its last successful request that its new request will be just under the rate limit and succeed. In some environments, this is fine and desired. However, my current goal is to firmly cut off clients that are making requests too fast, so I don't want this; instead, I implemented the 'strict' behavior so you don't get through at all until your request rate and the interval since your last request drops low enough.

Mathematically, a client that makes requests with little or no gap between them (to the precision of your timestamps) can wind up increasing its rate by slightly over its 'cost' per request. If I'm understanding the math correctly, how much over the cost is capped by Tony Finch's 'max(interval, 1.0e-10)' step, with 1.0e-10 being a small but non-zero number that you can move up or down depending on, eg, your language and its floating point precision. Having looked at it, in Python the resulting factor with 1.0e-10 is '1.000000082740371', so you and I probably don't need to worry about this. If the client doesn't make requests quite that fast, its rate will go up each time by slightly less than the 'cost' you've assigned. In Python, a client that makes a request every millisecond has a factor for this of '0.9995001666249781' of the cost; slower request rates make this factor smaller.

This is probably mostly relevant if you're dumping or reporting the calculated rates (for example, when a client hits the rate limit) and get puzzled by the odd numbers that may be getting reported.

In theory I have a formal education as a programmer (although it was a long time ago). In practice my knowledge from it isn't comprehensive, and every so often I run into an area where I know there's relevant knowledge and algorithms but I don't know what they are and I'm not sure how to find them. Today's area is scalable rate-limiting with low storage requirements.

Suppose, not hypothetically, that you want to ratelimit a collection of unpredictable sources and not use all that much storage per source. One extremely simple and obvious approach is to store, for each source, a start time and a count. Every time the source makes a request, you check to see if the start time is within your rate limit interval; if it is, you increase the count (or ratelimit the source), and if it isn't, you reset the start time to now and the count to 1.

(Every so often you can clean out entries with start times before your interval.)

The disadvantage of this simple approach is that it completely forgets about the past history of each source periodically. If your rate limit intervals are 20 minutes, a prolific source gets to start over from scratch every 20 minutes and run up its count until it gets rate limited again. Typically you want rate limiting not to forget about sources so fast.

I know there are algorithms that maintain decaying averages or moving (rolling) averages. The Unix load average is maintained this way, as is Exim ratelimiting. The Unix load average has the advantage that it's updated on a regular basis, which makes the calculation relatively simple. Exim has to deal with erratic updates that are unpredictable intervals from the previous update, and the comment in the source is a bit opaque to me. I could probably duplicate the formula in my code but I'd have to do a bunch of work to convince myself the result was correct.

(And now I've found Tony Finch's exponential rate limiting (via), which I'm going to have to read carefully, along with the previous GCRA: leaky buckets without the buckets.)

Given that rate limiting is such a common thing these days, I suspect that there are a number of algorithms for this with various different choices about how the limits work. Ideally, it would be possible to readily find writeups of them with internet searches, but of course as you know internet search is fairly broken these days.

(For example you can find a lot of people giving high level overviews of rate limiting without discussing how to actually implement it.)

Now that I've found Tony Finch's work I'm probably going to rework my hacky rate limiting code to do things better, because my brute force approach is using the same space as leaky buckets (as covered in Tony Finch's article) with inferior results. This shows the usefulness of knowing algorithms instead of just coding away.

(Improving the algorithm in my code will probably make no practical difference, but sometimes programming is its own pleasure.)

Over on the Fediverse, I said something that has a story attached:

It sounds obvious to say it, but I need to remember that I can always switch buffers in GNU Emacs by just switching buffers, not by using, eg, the MH-E commands to switch (back) to another folder. The MH-E commands quite sensibly do additional things, but sometimes I don't want them.

GNU Emacs has a spectrum of things that range from assisting your conventional editing (such as LSP clients) to what are essentially nearly full-blown applications that happen to be embedded in GNU Emacs, such as magit and MH-E and the other major modes for reading your email (or Usenet news, or etc). One of my personal dividing lines is to what extent the mode takes over from regular Emacs keybindings and regular Emacs behaviors. On this scale, MH-E is quite high on the 'application' side; in MH-E folder buffers, you mostly do things through custom keybindings.

(Well, sort of. This is actually overselling the case because I use regular Emacs buffer movement and buffer searching commands routinely, and MH-E uses Emacs marks to select ranges of messages, which you establish through normal Emacs commands. But actual MH-E operations, like switching to another folder, are done through custom keybindings that involve MH-E functions.)

My dominant use of GNU Emacs at the moment is as a platform for MH-E. When I'm so embedded in an MH-E mindset, it's easy to wind up with a form of tunnel vision, where I think of the MH-E commands as the only way to do something like 'switch to another (MH) folder'. Sometimes I do need or want to use the MH-E commands, and sometimes they're the easiest way, but part of the power of GNU Emacs as a general purpose environment is that ultimately, MH-E's displays of folders and messages, the email message I'm writing, and so on, are all just Emacs buffers being displayed in Emacs windows. I don't have to switch between these things through MH-E commands if I don't want to; I can just switch buffers with 'C-x b'.

(Provided that the buffer already exists. If the buffer doesn't exist, I need to use the MH-E command to create it.)

Sometimes the reason to use native Emacs buffer switching is that there's no MH-E binding for the functionality, for example to switch from a mail message I'm writing back to my inbox (either to look at some other message or to read new email that just came in). Sometimes it's because, for example, the MH-E command to switch to a folder wants to rescan the MH folder, which forces me to commit or discard any pending deletions and refilings of email.

One of the things that makes this work is that MH-E uses a bunch of different buffers for things. For example, each MH folder gets its own separately named buffer, instead of MH-E simply loading the current folder (whatever it is) into a generic 'show a folder' buffer. Magit does something similar with buffer naming, where its summary buffer isn't called just 'magit' but 'magit: <directory>' (I hadn't noticed that until I started writing this entry, but of course Magit would do it that way as a good Emacs citizen).

Now that I've written this, I've realized that a bit of my MH-E customization uses a fixed buffer name for a temporary buffer, instead of a buffer name based on the current folder. I'm in good company on this, since a number of MH-E status display commands also use fixed-name buffers, but perhaps I should do better. On the other hand, using a fixed buffer name does avoid having a bunch of these buffers linger around just because I used my command.

(This is using with-output-to-temp-buffer, and a lot of use of it in GNU Emacs' standard Lisp is using fixed names, so maybe my usage here is fine. The relevant Emacs Lisp documentation doesn't have style and usage notes that would tell me either way.)

GNU Emacs is famous as an editor written largely in itself, well, in Emacs Lisp, with a C core for some central high performance things and things that have to be done in C (called 'primitives' in Emacs jargon). It's perhaps popular to imagine that the overall structure of this is that the C parts of GNU Emacs expose a minimal and direct API that's mostly composed of primitive operations, so that as much of Emacs as possible can be implemented in Emacs Lisp. Unfortunately, this isn't really the case, or at least not necessarily as you'd like it, and one consequence of this is to limit the amount of customization you can feasibly do to GNU Emacs.

An illustration of this is in how GNU Emacs de-iconifies frames in X. In a minimal C API version of GNU Emacs, there might be various low level X primitives, including 'x-deiconify-frame', and the Emacs Lisp code for frame management would call these low level X primitives when running under X, and other primitives when running under Windows, and so on. In the actual GNU Emacs, deiconification of frames happens at multiple points and the exposed primitives are things like raise-frame and make-frame-visible. As their names suggest, these primitives aren't there to give Emacs Lisp code access to low level X operations, they're there to do certain higher level logical things.

This is a perfectly fair and logical decision by the GNU Emacs developers. To put it one way, GNU Emacs is opinionated. It and its developers have a certain model of how it works and how things should behave, what it means for the program to be 'GNU Emacs' as opposed to a hypothetical editor construction kit, and what the C code does is a reflection of that. To the Emacs developers, 'make a frame visible' is a sensible thing to do and is best done in C, so they did it that way.

(Buffers are another area where Emacs is quite opinionated on how it wants to work. This sometimes gets awkward, as anyone who's wrestled with temporarily displaying some information from Emacs Lisp may have experienced.)

The drawback of this is that sometimes you can only easily customize GNU Emacs in ways that line up with how the developers expected, since you can't change the inside of C level primitives. If your concept of an operation you want to hook, modify, block, or otherwise fiddles with matches with how GNU Emacs sees things, all is probably good. But if your concept of 'an operation' doesn't match up with how GNU Emacs sees it, you may find that what you want to touch is down inside the C layer and isn't exposed as a separate primitive.

(Even if it is exposed as a primitive in its own right, you can have problems, because when you advise a primitive, this doesn't affect calls to the primitive from other C functions. If there was a separate 'x-deiconify-frame' primitive, I could hook it for calls from Lisp, but not a call from 'make-frame-visible' if that was still a primitive. So to really have effective hooking of a primitive, you need it to be only called from Lisp code (at least for cases you care about).)

PS: This can lead to awkward situations even when everything you want to modify is in Emacs Lisp code, because the specific bit you want to change may be in the middle of a large function. Of course with Emacs Lisp you can always redefine the function, copying its code and modifying it to taste, but there are still drawbacks. You can make this somewhat more reliable in the face of changes (via a comment on this entry, but it's still not great.

Recently I saw this example Go code (via), and I had to stare at it a while in order to understand what it was doing and how it worked (and why it had to be that way). The goal of waitReadAll() is to either receive (read) all currently available items from a channel (possibly a buffered one) or to time out if nothing shows up in time. This requires two nested selects, with the inner one in a for loop.

The outer select has this form:

select {
  case v, ok := <- c:
    if !ok {
      return ...
    }
    [... inner code ...]

  case <- time.After(dur) // wants go 1.23+
    return ...
}

This is doing three things. First (and last in the code), it's timing out of the duration expires before anything is received on the channel. Second, it's returning right away if the channel is closed and empty; in this case the channel receive from c will succeed, but ok will be false. And finally, in the code I haven't put in, it has received the first real value from the channel and now it has to read the rest of them.

The job of the inner code is to receive any (additional) currently ready items from the channel but to give up if the channel is closed or when there are no more items. It has the following form (trimmed of the actual code to properly accumulate things and so on, see the playground for the full version):

.. setup elided ..
for {
  select {
    case v, ok := <- c:
      if ok {
        // accumulate values
      } else {
        // channel closed and empty
        return ...
      }
    case default:
      // out of items
      return ...
  }
}

There's no timeout in this inner code because the 'case default' means that we never wait for the channel to be ready; either the channel is ready with another item (or it's been closed), or we give up.

One of the reasons this Go code initially confused me is that I started out misreading it as receiving as much as it could from a channel until it reached a timeout. Code that did that would do a lot of the same things (obviously it needs a timeout and a select that has that as one of the cases), and you could structure it somewhat similarly to this code (although I think it's more clearly written without a nested loop).

(This is one of those entries that I write partly to better understand something myself. I had to read this code carefully to really grasp it and I found it easy to mis-read on first impression.)

Over on the Fediverse I had a long standing GNU Emacs gripe:

I would rather like to make it so that GNU Emacs never un-iconifies itself when it completes (Lisp-level) actions. If I have Emacs iconified I want it to stay that way, not suddenly appear under my mouse cursor like an extremely large modal popup. (Modal popups suck, they are a relic of single-tasking windowing environments.)

For those of you who use GNU Emacs and have never been unlucky enough to experience this, if you start some long operation in GNU Emacs and then decide to iconify it to get it out of your face, a lot of the time GNU Emacs will abruptly pop itself back open when it finishes, generally with completely unpredictable timing so that it disrupts whatever else you switched to in the mean time.

(This only happens in some X environments. In others, the desktop or window manager ignores what Emacs is trying to do and leaves it minimized in your taskbar.)

To cut straight to the answer, you can avoid a lot of this with the following snippet of Emacs Lisp:

(add-to-list 'display-buffer-alist '(t nil (inhibit-switch-frame . t)))

I believe that this has some side effects but that these side effects will generally be that Emacs doesn't yank around your mouse focus or suddenly raise windows to be on top of everything.

GNU Emacs doesn't have a specific function that it calls to de-iconify a frame, what Emacs calls a top level window. Instead, the deiconification happens in C code inside C-level functions like raise-frame and make-frame-visible, which also do other things and which are called from many places. For instance, one of make-frame-visible's jobs is actually displaying the frame's X level window if it doesn't already exist on the screen.

(There's an iconify-or-deiconify-frame function but if you look that's a Lisp function that calls make-frame-visible. It's only used a little bit in the Emacs Lisp code base.)

A determined person could probably hook these C-level functions through advice-add to make them do nothing if they were called on an existing, mapped frame that was just iconified. That would be the elegant way to do what I want. The inelegant way is to discover, via use of the Emacs Lisp debugger, that everything I seem to care about is going through 'display-buffer' (eventually calling window--maybe-raise-frame), and that display-buffer's behavior can be customized to not 'switch frames', which will wind up causing things to not call window--maybe-raise-frame and not de-iconify GNU Emacs windows on me.

To understand display-buffer-alist I relied on Demystifying Emacs’s Window Manager. My addition to display-buffer-alist has three elements:

• the t tells display-buffer to always use this alist entry.
• the nil tells display-buffer that I don't have any special action functions I want to use here and it should just use its regular ones. I think an empty list might be more proper here, but nil works.
• the '(inhibit-switch-frame . t)' sets the important customization, which will be merged with any other things set by other (matching) alist entries.

The net effect is that 'display-buffer' will see 'inhibit-switch-frame' set for every buffer it's asked to switch to, and so will not de-iconify, raise, or otherwise monkey around with frame things in the process of displaying buffers. It's possible that this will have undesirable side effects in some circumstances, but as far as I can tell things like 'speedbar' and 'C-x 5 <whatever>' still work for me afterward, so new frames are getting created when I want them to be.

(I could change the initial 't' to something more complex, for example to only apply this to MH-E buffers, which is where I mostly encounter the problem. See Demystifying Emacs’s Window Manager for a discussion of how to do this based on the major mode of the buffer.)

To see if you're affected by this, you can run the following Emacs Lisp in the scratch buffer and then immediately minimize or iconify the window.

(progn
  (sleep-for 5)
  (display-buffer "*scratch*"))

If you're affected, the Emacs window will pop back open in a few seconds (five or less, depending on how fast you minimized the window). If the Emacs window stays minimized or iconified, your desktop environment is probably overriding whatever Emacs is trying to do.

For me this generally happens any time some piece of Emacs Lisp code is taking a long time to get a buffer ready for display and then calls 'display-buffer' at the end to show the buffer. One trigger for this is if the buffer to be displayed contains a bunch of unusual Unicode characters (possibly ones that my font doesn't have anything for). The first time the characters are used, Emacs will apparently stall working out how to render them and then de-iconify itself if I've iconified it out of impatience.

(It's quite possible that there's a better way to do this, and if so I'd love to know about it.)

Suppose that you have a shell script that's not necessarily complex but is at least long. For reliability, you use 'set -e' so that the script will immediately stop on any unexpected errors from commands, and sometimes this happens. Since this isn't supposed to happen, it would be nice to print some useful information about what went wrong, such as where it happened, what the failing command's exit status was, and what the command was. The good news is that if you're willing to make your script specifically a Bash script, you can do this quite easily.

The Bash trick you need is:

trap 'echo "Exit status $? at line $LINENO from: $BASH_COMMAND"' ERR

This uses three Bash features: the special '$LINENO' and '$BASH_COMMAND' environment variables (which have the command executed just before the trap and the line number), and the special 'ERR' Bash 'trap' condition that causes your 'trap' statement to be invoked right when 'set -e' is causing your script to fail and exit.

Using 'ERR' instead of 'EXIT' (or '0' if you're a traditionalist like me) is necessary in order to get the correct line number in Bash. If you switch this to 'trap ... EXIT', the line number that Bash will report is the line that the 'trap' was defined on, not the line that the failing command is on (although the command being executed remains the same). This makes a certain amount of sense from the right angle; the shell is currently on that line as it's exiting.

As far as I know, no other version of the Bourne shell can do all of this. The OpenBSD version of /bin/sh has a '$LINENO' variable and 'trap ... 0' preserves its value (instead of resetting it to the line of the 'trap'), but it has no access to the current command. The FreeBSD version of /bin/sh resets '$LINENO' to the line of your 'trap ... 0', so the best you can do is report the exit status. Dash, the Ubuntu 24.04 default /bin/sh, doesn't have '$LINENO', effectively putting you in the same situation as FreeBSD.

(On Fedora, /bin/sh is Bash, and the Fedora version of Bash supports all of 'trap .. ERR', $LINENO, and $BASH_COMMAND even when invoked as '#!/bin/sh' by your script. You probably shouldn't count on this; if you want Bash, use '#!/bin/bash'.)

One of the GNU Emacs packages that I use is which-key (which is now in GNU Emacs v30). I use it because I don't always remember specific extended key bindings (ones that start with a prefix) that are available to me, especially in MH-E, which effectively has its own set of key bindings for a large collection of commands. Which-key gives me a useful but minimal popup prompt about what's available and I can usually use that to navigate to what I want. Recently I read Omar Antolín's The case against which-key: a polemic, which didn't convince me but did teach me two useful tricks in one.

The first thing I learned from it, which I could have already known if I was paying attention, was the Ctrl-h keybinding that's available in extended key bindings to get some additional help. In stock GNU Emacs this Ctrl-h information is what I consider only somewhat helpful, and with which-key turned on it's basically not helpful at all from what I can see (which may partly explain why I didn't pay it any attention before).

The second thing is a way to make this Ctrl-h key binding useful in combination with which-key, using embark and in fact I believe a number of my other minibuffer completion things. That is to bind Ctrl-h to an Embark function designed for this:

(setq prefix-help-command #'embark-prefix-help-command)

As illustrated in the article, using Ctrl-h with this binding effectively switches your multi-key entry over to minibuffer completion, complete with all of the completion add-ons you have configured. If you've got things like Vertico, Marginalia, and Orderless configured, this is an excellent way to pick through the available bindings to figure out what you want; with my configuration I get the key itself, the ELisp function name, and the ELisp help summary (and then I can cursor down and up through the list).

The Embark version is too much information if I just need a little reminder of what's possible; that's what the basic which-key display is great for. But if the basic information isn't enough, the Embark binding for Ctrl-h is a great supplement, and it even reaches through multi-key sequences (which is something that which-key doesn't do, at least in my setup, and I have some of them in MH-E).

One of the reactions I saw to my entry on malloc(0) being allowed to return NULL is to wonder why you'd ever do a zero-size allocation in the first place. Unfortunately, it's relatively easy to stumble into this situation with simple code in certain sorts of not particularly uncommon situations. The most obvious one is if you're allocating memory for a variable size object, such as a Python tuple or a JSON array. In a simple C implementation these will typically have a fixed struct that contains a pointer to a C memory block with either the actual elements or an array of pointers to them. The natural way to set this up is to write code that winds up calling 'malloc(nelems * sizeof(...))' or something like that, like this:

array_header *
alloc_array(unsigned int nelems)
{
   array_header *h;
   h = malloc(sizeof(array_header))
   if (h == NULL) return NULL;

   /* get space for the element pointers except oops */
   h->data = malloc(nelems * sizeof(void *));
   if (h->data == NULL) {
      free(h);
      return NULL;
   }

   h->nelems = nelems;
   /* maybe some other initialization */

   return h;
}

(As a disclaimer, I haven't tried to compile this C code because I'm lazy, so it may contain mistakes.)

Then someone asks your code to create an empty tuple or JSON array and on some systems, things will explode because nelems will be 0 and you will wind up doing 'malloc(0)' and that malloc() will return NULL, as it's allowed to, and your code will think it's out of memory. You can obviously prevent this from happening, but it requires more code and thus requires you to have thought of the possibility.

(Allocating C strings doesn't have this problem because you always need one byte for the terminating 0 byte, but it can come up with other forms of strings where you track the length explicitly.)

One tricky bit about this code is that it will only ever go wrong in an obvious way on some uncommon systems. On most systems today, 'malloc(0)' returns a non-NULL result, usually because the allocator rounds up the amount of memory you asked for to some minimum size. So you can write this code and have it pass all of your tests on common platforms and then some day someone reports that it fails on, for example, AIX.

(It's possible that modern C linters and checkers will catch this; I'm out of touch with the state of the art there.)

As a side note, if malloc(0) returns anything other than NULL, I believe that each call is required to return a unique pointer (see eg the POSIX description of malloc()). I believe that these unique pointers don't have to point to actual allocated memory; they could point to some large reserved arena and be simply allocated in sequence, with a free() of them effectively doing nothing. But it's probably simpler to have your allocator round the size up and return real allocated memory, since then you don't have handle things like the reserved arena running out of space.

One of the little traps in standard C and POSIX is that malloc(0) is allowed to return NULL instead of a pointer. This makes people unhappy for various reasons. Today I wound up reading 017. malloc(0) & realloc(…, 0) ≠ 0, which runs through a whole collection of Unix malloc() versions and finds almost none of them which return NULL on malloc(0) except for some Unix System V releases that ship with an optional 'fast' malloc library that does return NULL on zero-sized allocations. Then AT&T wrote the System V Interface Definition and requires this 'fast malloc' behavior, except that actual System V releases (probably) didn't behave this way unless you explicitly used the fast malloc instead of the standard one.

(Apparently AIX may behave this way, eg, and it's old enough to have influenced POSIX and C. But I suspect that AIX got this behavior by making the System V fast malloc their only malloc, possibly when the SVID nominally required this behavior. AIX may have wound up weird but IBM didn't write it from scratch.)

When I read all of this today and considered what POSIX had done, one of my thoughts was about non-Unix C compilers (partly because I'd recently heard about the historical Whitesmiths C compiler source code being released). C was standardized at a time when C was being increasingly heavily used on various personal computers, including in environments that were somewhat hostile to it, and also other non-Unix environments. These C implementations used their own standard libraries, including malloc(), so maybe they had adopted the NULL return behavior.

As far as I can tell, Whitesmiths' malloc() doesn't have this behavior (also). However, I did find this in the MS-DOS version of Manx Aztec C, or at least it's in version 5.2a; the two earlier versions also available have a simpler malloc() that always rounds up, like the Whitesmiths malloc(). My memory is that you could get the Manx Aztec C compiler for the Amiga with library source, but I'm not particularly good at poking around the Amiga image available so I was unable to spot it if it's included in that version, and I haven't looked at the other Aztec C versions.

(I wouldn't be surprised if a number of 1980s non-Unix C compilers had this behavior, but I don't know where to find good information on this. If someone has written a comprehensive history page on malloc(0) that covers non-Unix C compilers, I haven't found it.)

On systems with small amounts of memory, one reason to specifically make your malloc() return NULL for 0-sized allocations is to reduce memory usage if someone makes a number of such allocations through some general code path that deals with variable-sized objects. Otherwise you'd have to consume some minimum amount of memory even for these useless allocations.

PS: Minix version 1 also rounds up the size of malloc(0).

(Yes, I got nerd-sniped by this and my own curiosity.)

Recently I read [ On ¦ No ] syntactic support for error handling, where the Go team have decided that they're not going to do anything with error handling (contrary to my fears). One part of it sparked a thought about making it less annoying to write basic, boilerplate error handling, and I briefly said something on the Fediverse:

A thought on Go error handling: you could reduce the annoyance of writing the boiler plate by creating a code formatter (along the lines of goimports) that added the standard 'if (err != nil) {...}' stuff to any unhandled 'err' return in your code. Then you'd write code with just 'a, err := Thing(); b, err := Thing2(a)' and so on, and the code formatter would fill in things for you.

This thought was inspired by the model of goimports; my impression is that almost everyone uses goimports to automatically update import statements as part of (auto-)formatting their code (certainly I do). If goimports can update imports as part of 'formatting', then in theory we can extend it to add boilerplate 'check and return' error handling. This would take a set of code like this:

a, err := Thing()
b, err := Thing2(a)
return Thing3(b)

This code would be turned into:

a, err := Thing()
if (err != nil) {
  return <zero value>, err
}
b, err := Thing2(a)
if (err != nil) {
  return <zero value>, err
}
return Thing3(b)

(Here by '<zero value>' I mean some suitable zero value, like 'nil', not the literal text '<zero value>'.)

This would only happen if the error return was assigned to a variable and then the variable was unused (before potentially being reassigned). If you wrote 'a, _ := Thing()', there would be no error check added, and if you re-ran the formatter on the post-formatter code it wouldn't do anything because all of the error variables are used. Determining if the error variable was used would need some control flow analysis, and determining what to return and the syntax of creating appropriate zero values for non-error return values would take some function signature and type analysis.

(As an extra trick, if you had an 'if (err != nil) {return ...}' block already, the formatter could copy the 'return ...' into the blocks it added.)

A Go code formatter isn't the only place you could implement this feature. These days many people would be able to use it if it was a 'code action' in a LSP server such as gopls, which already supports other code actions. As a LSP server code action, you could easily apply it selectively only to a function (or a section of a function), rather than having to trust the formatter to run over your entire file. You could also have a LSP server code action that added the boilerplate error check immediately after a call, so you'd write the function call line and then pick 'add boilerplate error check' at the end.

I don't know if this is a good idea, either as a standalone formatter or as a LSP code feature. I'm not a big user of gopls code actions (I mostly ignore them) and I'm not sure I'd be happy writing code that looked like the starting point. But perhaps some people would be, especially if it was an 'expand this function call to have boilerplate error checks' in gopls.

(I don't write enough Go code to have strong feelings about the boilerplate error handling. At my scale, hand-writing the standard 'if (err != nil)' stuff is fine, and I not infrequently want to do something with the error.)

As covered in my Emacs packages, vertico is one of the third party Emacs packages that I have installed to modify how minibuffer completion works for me, or at least how it looks. In my experience, vertico took a significant amount of customization before I really liked it (eventually including some custom code), so I'm going to write down some notes about why I made various settings.

Vertico itself is there to always show me a number of the completion targets, as a help to narrowing in on what I want; I'm willing to trade vertical space during completion for a better view of what I'm navigating around. It's not the only way to do this (there's fido-vertical-mode in standard GNU Emacs, for example), but it's what I started with and it has a number of settings that let me control both how densely the completions are presented (and so how many of them I get to see at once) and how they're presented.

The first thing I do with vertico is override its key binding for TAB, because I want standard Emacs minibuffer tab completion, not vertico's default behavior of inserting the current thing completion is currently on. Specifically, my key bindings are:

 :bind (:map vertico-map
             ("TAB" . minibuffer-complete)
             ;; M-v is taken by vertico
             ("M-g M-c" . switch-to-completions)
             ;; Original tab binding, which we want sometimes when
             ;; using orderless completion.
             ("M-TAB" . vertico-insert))

I normally work by using regular tab completion and orderless's completion until I'm happy, then hitting M-TAB if necessary and then RET. I use M-g M-c so rarely that I'd forgotten it until writing this entry. Using M-TAB is especially likely for a long filename completion, where I might use the cursor keys (or theoretically the mouse) to move vertico's selection to a directory and then hit M-TAB to fill it in so I can then tab-complete within it.

Normally, vertico displays a single column of completion candidates, which potentially leaves a lot of wasted space on the right; I use marginalia to add information some sorts of completion targets (such as Emacs Lisp function names) in this space. For other sorts of completions where there's no particular additional information, such as MH-E mail folder names, I use vertico's vertico-multiform-mode to switch to a vertico-grid so I fill the space with several columns of completion candidates and reduce the number of vertical lines that vertico uses (both are part of vertico's extensions).

(I also have vertico-mouse enabled when I'm using Emacs under X, but in practice I mostly don't use it.)

Another important change (for me) is to turn off vertico's default behavior of remembering the history of your completions and putting recently used entries first in the list. This sounds like a fine idea, but in practice I want my completion order to be completely predictable and I'm rarely completing the same thing over and over again. The one exception is my custom MH-E folder completion, where I do enable history because I may be, for example, refiling messages into one of a few folders. This is done through another extension, vertico-sort, or at least I think it is.

(When vertico is installed as an ELPA or MELPA package and then use-package'd, you apparently get all of the extensions without necessarily having to specifically enable them and can just use bits from them.)

My feeling is that effective use of vertico probably requires this sort of customization if you regularly use minibuffer completion for anything beyond standard things where vertico (and possibly marginalia) can make good use of all of your horizontal space. Beyond what key bindings and other vertico behavior you can stand and what behavior you have to change, you want to figure out how to tune vertico so that it's significantly useful for each thing you regularly complete, instead of mostly showing you a lot of empty space and useless results. This is intrinsically a relatively personal thing.

PS: One area where vertico's completion history is not as useful as it looks is filename completion or anything that looks like it (such as standard MH-E folder completion). This is because Emacs filename completion and thus vertico's history happens component by component, while you probably want your history to give you the full path that you wound up completing.

PPS: I experimented with setting vertico-resize, but found that the resulting jumping around was too visually distracting.

Suppose, not hypothetically, that you have a shell script with a relatively simple configuration file format that people can still accidentally get wrong. You'd like to check the configuration file for problems before you use it in the rest of your script, for example by using it with 'join' (where things like the wrong number or type of fields will be a problem). Recently on the Fediverse I shared how I was doing this with awk, so here's a slightly more elaborate and filled out version:

errs=$(awk '
         $1 ~ "^#" { next }
         NF != 3 {
            printf " line %d: wrong number of fields\n", NR;
            next }
         [...]
         ' "$cfg_file"
       )

if [ -n "$errs" ]; then
   echo "$prog: Errors found in '$cfg_file'. Stopping." 1>&2
   echo "$errs" 1>&2
   exit 1
fi

(Here I've chosen to have awk's diagnostic messages indented by one space when the script prints them out, hence the space before 'line %d: ...'.)

The advantage of having awk simply print out the errors it detects and letting the script deal with them later is that you don't need to mess around with awk's exit status; your awk program can simply print what it finds and be done. Using awk for the syntax checks is handy because it lets you express a fair amount of logic and checks relatively simply (you can even check for duplicate entries and so on), and it also gives you line numbers for free.

One trick with using awk in this way is to progressively filter things in your checks (by skipping further processing of the current line with 'next'). We start out by skipping all comments, then we report and otherwise skip every line with the wrong number of fields, and then every check after this can assume that at least we have the right number of fields so it can confidently check what should be in each one. If the number of fields in a line is wrong there's no point in complaining about how one of them has the wrong sort of value, and the early check and 'next' to skip the rest of this line's processing is the simple way.

If you're also having awk process the configuration file later you might be tempted to have it check for errors at the same time, in an all-in-one awk program, but my view is that it's simpler to split the error checking from the processing. That way you don't have to worry about stopping the processing if you detect errors or intermingle processing logic with checking logic. You do have to make sure the two versions have the same handling of comments and so on, but in simple configuration file formats this is usually easy.

(Speaking from personal experience, you don't want to use '$1 == "#"' as your comment definition, because then you can't just stick a '#' in front of an existing configuration file line to comment it out. Instead you have to remember to make it '# ', and someday you'll forget.)

PS: If your awk program is big and complex enough, it might make more sense to use a here document to create a shell variable containing it, which will let you avoid certain sorts of annoying quoting problems.

Firefox recently officially switched to Git, in a completely different Git tree than their old mirror. This presented me a little bit of a problem because I have a collection of local changes I make to my own Firefox builds, which I carry as constantly-rebased commits on top of the upstream Firefox tree. The change in upstream trees meant that I was going to have to move my commits to the new tree. When I wrote my first entry I thought I might try to do this in some clever way similar to rebasing my own changes on top of something that was rebased, but in the end I decided to do it the simple and brute force way that I was confident would either work or would leave me in a situation I could back out from easily.

This simple and brute force way was to get both my old tree and my new 'firefox' tree up to date, then export my changes with 'git format-patch' from the old tree and import them into the new tree with 'git am'. There were a few irritations along the way, of course. First I (re)discovered that 'git am' can't directly consume the directory of patches you create with 'git format-patch'. Git-am will consume a Maildir of patches, but git-format-patch will only give you a directory full of files with names like '00NN-<author>-<title>.patch', which is not a proper Maildir. The solution is to cat all of the .patch files together in order to some other file, which is now a mailbox that git-am will handle. The other minor thing is that git-am unsurprisingly has no 'dry-run' option (which would probably be hard to implement). Of course in my situation, I can always reset 'main' back to 'origin/main', which was one reason I was willing to try this.

(Looking at the 'git format-patch' manual page suggests that what I might have wanted was the '--stdout' option, which would have automatically created the mbox format version for me. On the other hand it was sort of nice to be able to look at the list of patches and see that they were exactly what I expected.)

On the one hand, moving my changes in this brute force way (and to a completely separate new tree) feels like giving in to my unfamiliarity with git. There are probably clever git ways to do this move in a single tree without having to turn everything into patches and then apply them (even if most of that is automated). On the other hand, this got the job done with minimal hassles and time consumed, and sometimes I need to put a stop to my programmer's urge to be clever.

Firefox's official source repository has moved to Git, but to a completely new Git repository, not the Git mirror that I've used for the past few years. This led me to a lament on the Fediverse:

This is my sad face that Firefox's switch to using git of course has completely different commit IDs than the old not-official gecko-dev git repository, meaning that I get to re-clone everything from scratch (all ~8 GB of it). Oh well, so it goes in the land of commit hashes.

Then Tim Chase pointed out something that I should have thought of:

If you add the new repo as a secondary remote in your existing one and pull from it, would it mitigate pulling all the blobs (which likely remain the same), limiting your transfer to just the commit-objects (and possibly some treeish items and tags)?

Git is famously a form of content-addressed storage, or more specifically a tree of content addressed storage, where as much as possible is kept the same over time. This includes all the portions of the actual source tree. A Git commit doesn't directly include a source tree; instead it just has the hash of the source tree (well, its top level, cf).

What this means is that if you completely change the commits so that all of them have new hashes, for example by rebuilding your history from scratch in a new version of the repository, but you keep the actual tree contents the same in most or all of the commits, the only thing that actually changes is the commits. If you add this new repository (with its new commit history) as a Git remote to your existing repository and pull from it, most or all of the tree contents are the same across the two sets of commits and won't have to be fetched. So you don't fetch gigabytes of tree contents, you only fetch megabytes (one hopes) of commits.

As I mentioned on the Fediverse, I was told this too late to save me from re-fetching the entire new Firefox repository from scratch on my office desktop (which has lots of bandwidth). I may yet try this on my home desktop, or alternately use it on my office desktop to easily move my local changes on top of the new official Git history.

(I think this is effectively rebasing my own changes on top of something that's been rebased, which I've done before, although not recently. I'll also want to refresh my understanding of what 'git rebase' does.)

One of the traditional divides in unit testing is whether you should write 'glass box' or 'opaque box' tests (like GeePawHill I think I prefer those terms to the traditional ones), which is to say whether you should write tests exploiting your knowledge of the module's code or without it. Since I prefer testing inside my modules, I'm implicitly on the side of glass box tests; even if I'm testing public APIs, I write tests with knowledge of potential corner cases. Recently, another reason for this occurred to me, by analogy to standards.

I've read about standards (and read the actual standards) enough by now to have absorbed the lesson that it is very hard to write a (computer) standard that can't be implemented perversely. Our standards need good faith implementations and there's only so much you can do to make it hard for people implementing them in bad faith. After that, you have to let the 'market' sort it out (including the market of whether or not people want to use perverse implementations, which generally they don't).

(Of course some time the market doesn't give you a real choice. Optimizing C compilers are an example, where your only two real options (GCC and LLVM) have aggressively exploited arguably perverse readings of 'undefined behavior' as part of their code optimization passes. There's some recent evidence that this might not always be worth it [PDF], via.)

If you look at them in the right way, unit tests are also a sort of standard. And like standards, opaque box unit tests have a very hard time of completely preventing perverse implementations. While people usually don't deliberately create perverse implementations, they can happen by accident or by misunderstandings, and there can be areas of perverse problems due to bugs. Your cheapest assurance that you don't have a perverse implementation is to peer inside and then write glass box tests that in part target the areas where perverse problems could arise. If you write opaque box tests, you're basically hoping that you can imagine all of the perverse mistakes that you'll make.

(Some things are amenable to exhaustive testing, but usually not very many.)

PS: One way to get perverse implementations is 'write code until all of the tests pass, then stop'. This doesn't guarantee a perverse implementation but it certainly puts the onus on the tests to force the implementation to do things, much like with standards (cf).

I've said in the past that Go is not our language, and I still stand by that. At the same time, the Go developers do eventually respond to the clamour from the community, which I maintain that we've seen with both Go's eventual addition of generics and the change to Go modules and Go dependency handling (where Go started with one story until it clearly didn't work and they had to change). This leads me to two related views.

First, I think that changes to Go's error handling are inevitably coming sooner or later. Error handling is something the community keeps being unhappy about (even though some people are fine with the current situation), and we know that some people in the core team have written up ideas (via, also). This issue is on the radar, and because it's such a popular issue, I think that change is inevitable.

At the same time, I'm not optimistic about that change, because I don't think error handling is a solved problem. We have a relatively good understanding of things like generics and dependency management, but we don't have a similar understanding of 'good error handling' that can drive a good new implementation in Go. It's possible that the Go developers will find something great, but I think it's more likely that what we'll get is a change that comes with its own set of drawbacks (although it'll be better overall than the current approach).

Go is slowly but steadily becoming a more and more complicated language, and a new method of error handling will inevitably add to that complexity. Also, as I once wrote about an earlier error handling proposal (and another one), a change in error handling will inevitably change how Go is written. People will be pushed to write code that works well with the new error handling mechanism and some number of people will use it for nominally clever tricks, because that's what happens with any language feature.

All of this leaves me feeling somewhat pessimistic about any error handling changes to Go. The current situation isn't ideal, but at least the language is kept simple. Given that error handling isn't a solved problem, I'm not sure any error handling change will improve things enough to make up for its other effects.

When I wrote about how error handling isn't a solved problem, I said some things about Rust's error handling that were flat out wrong, which I had in my mind through superstition. Today is a brief correction on that, since I looked it up.

Rust's usual way of signalling (recoverable) errors is to use the Result type, which is an enum with one option for errors and one option for success (so it is the Go 'result, err := call(...)' pattern where only one of result and err can be valid at once, and you have to check before using either). The verbose way of handling this is to explicitly match and handle each option. However, often you're only going to propagate an error, and Rust has special syntax for that in the '?' operator, which immediately propagates an error return and otherwise continues:

fn read_username_from_file() -> Result<String, io::Error> {
    let mut username_file = File::open("hello.txt")?;
    [...]

Rust's '?' operator can be used in any function with a compatible return type. This includes main() if you declare it appropriately, so you can use error propagation with '?' as your only way of handling errors all through your program if you want. The result will probably be a little bit mysterious since people won't get any specific message on error, just a non-zero exit status, but for quick programs I can see the appeal of doing this all the way up through main().

(This makes the '?' operator a far less verbose equivalent of the common Go idiom of 'r, err := ...; if err != nil {return ..., err}'. The '...' will vary depending on the function's return type.)

The other approach is to panic on error with .unwrap(), if you're okay with a basic panic, or .expect(), if you want to provide some sort of diagnostic message to explain a bit about the problem. Although Rust people will probably twitch at this comparison, it feels to me like using .unwrap() is the equivalent of a Python program that does nothing to catch any exceptions (and so winds up with the default stack backtrace), while .expect() is the equivalent of a Python try/except block that prints some sort of a message before exiting.

Since both of these approaches are using Result, you can combine them in quick utility programs. You can propagate errors upward through most of your code, then .expect() on high level operations in main() or functions directly below it to provide some information if things go wrong.

(As a sysadmin, I'm used to the idea of writing quick and rough programs that are run by hand, used only rarely, and operate in environments where they almost never expect to fail. These programs often can get away with minimal error handling, but if things do go wrong it's handy to have some idea of roughly what and where.)

Obviously, what I'd vaguely remembered was a common usage of .unwrap(). I think the wires got crossed in my mind because more recent Rust code I've seen uses '?' a lot, so I sort of vaguely crossed the two in my mind.

Yesterday I wrote about Go's use of multiple return values and Go types, in reaction to Mond's Were multiple return values Go's biggest mistake?. One of the things that I forgot to mention in that entry is that I think Go's choice to have multiple values for function returns and a few other things was the simpler and more conservative approach in its overall language design.

In a statically typed language that expects to routinely use multiple return values, as Go was designed to with the 'result, error' pattern, returning multiple values as a typed tuple means that tuple-based types are pervasive. This creates pressures on both the language design and the API of the standard library, especially if you start out (as Go did) being a fairly strongly nominally typed language, where different names for the same concrete type can't be casually interchanged. Or to put it another way, having a frequently used tuple container (meta-)type significantly interacts with and affects the rest of the language.

(For example, if Go had handled multiple values through tuples as explicit typed entities, it might have had to start out with something like type aliases (added only in Go 1.9) and it might have been pushed toward some degree of structural typing, because that probably makes it easier to interact with all of the return value tuples flying around.)

Having multiple values as a special case for function returns, range, and so on doesn't create anywhere near this additional influence and pressure on the rest of the language. There are a whole bunch of questions and issues you don't face because multiple values aren't types and can't be stored or manipulated as single entities. Of course you have to be careful in the language specification and it's not trivial, but it's simpler and more contained than going the tuple type route. I also feel it's the more conservative approach, since it doesn't affect the rest of the language as much as a widely used tuple container type would.

(As Mond criticizes, it does create special cases. But Go is a pragmatic language that's willing to live with special cases.)

Recently I read Were multiple return values Go's biggest mistake? (via), which wishes that Go had full blown tuple types (to put my spin on it). One of the things that struck me about Go's situation when I read the article is exactly the inverse of what the article is complaining about, which is that because Go allows multiple values for function return types (and in a few other places), it doesn't have to have tuple types.

One problem with tuple types in a statically typed language is that they must exist as types, whether declared explicitly or implicitly. In a language like Go, where type definitions create new distinct types even if the structure is the same, it isn't particularly difficult to wind up with an ergonomics problem. Suppose that you want to return a tuple that is a net.Conn and an error, a common pair of return values in the net package today. If that tuple is given a named type, everyone must use that type in various places; merely returning or storing an implicitly declared type that's structurally the same is not acceptable under Go's current type rules. Conversely, if that tuple is not given a type name in the net package, everyone is forced to stick to an anonymous tuple type. In addition, this up front choice is now an API; it's not API compatible to give your previously anonymous tuple type a name or vice versa, even if the types are structurally compatible.

(Since returning something and error is so common an idiom in Go, we're also looking at either a lot of anonymous types or a lot more named types. Consider how many different combinations of multiple return values you find in the net package alone.)

One advantage of multiple return values (and the other forms of tuple assignment, and for range clauses) is that they don't require actual formal types. Functions have a 'result type', which doesn't exist as an actual type, but you also needed to handle the same sort of 'not an actual type' thing for their 'parameter type'. My guess is that this let Go's designers skip a certain amount of complexity in Go's type system, because they didn't have to define an actual tuple (meta-)type or alternately expand how structs worked to cover the tuple usage case,

(Looked at from the right angle, structs are tuples with named fields, although then you get into questions of nested structs act in tuple-like contexts.)

A dynamically typed language like Python doesn't have this problem because there are no explicit types, so there's no need to have different types for different combinations of (return) values. There's simply a general tuple container type that can be any shape you want or need, and can be created and destructured on demand.

(I assume that some statically typed languages have worked out how to handle tuples as a data type within their type system. Rust has tuples, for example; I haven't looked into how they work in Rust's type system, for reasons.)

There are certain things about programming language design that are more or less solved problems, where we generally know what the good and bad approaches are. For example, over time we've wound up agreeing on various common control structures like for and while loops, if statements, and multi-option switch/case/etc statements. The syntax may vary (sometimes very much, as for example in Lisp), but the approach is more or less the same because we've come up with good approaches.

I don't believe this is the case with handling errors. One way to see this is to look at the wide variety of approaches and patterns that languages today take to error handling. There is at least 'errors as exceptions' (for example, Python), 'errors as values' (Go and C), and 'errors instead of results and you have to check' combined with 'if errors happen, panic' (both Rust). Even in Rust there are multiple idioms for dealing with errors; some Rust code will explicitly check its Result types, while other Rust code sprinkles '?' around and accepts that if the program sails off the happy path, it simply dies.

Update: I got Rust's error handling wrong, as pointed out in the comments on this entry. What I was thinking of is Rust's .unwrap() and .expect(), not '?'.

If you were creating a new programming language from scratch, there's no clear agreed answer to what error handling approach you should pick, not the way we have more or less agreed on how for, while, and so on should work. You'd be left to evaluate trade offs in language design and language ergonomics and to make (and justify) your choices, and there probably would always be people who think you should have chosen differently. The same is true of changing or evolving existing languages, where there's no generally agreed on 'good error handling' to move toward.

(The obvious corollary of this is that there's no generally agreed on keywords or other syntax for error handling, the way 'for' and 'while' are widely accepted as keywords as well as concepts. The closest we've come is that some forms of error handling have generally accepted keywords, such as try/catch for exception handling.)

I like to think that this will change at some point in the future. Surely there actually is a good pattern for error handling out there and at some point we will find it (if it hasn't already been found) and then converge on it, as we've converged on programming language things before. But I feel it's clear that we're not there yet today.

One of the things I do with Git is maintain personal changes locally on top of the upstream version, with my changes updated via rebasing every time I pull upstream to update it. In the simple case, I have only a single local change and commit, but in more complex cases I split my changes into multiple local commits; my local version of Firefox currently carries 12 separate personal commits. Every so often, upstream changes something that causes one of those personal changes to need an update, without actually breaking the rebase of that change. When this happens I need to update my local commit with more changes, and often it's not the 'top' local commit (which can be updated simply).

In theory, the third party tool git-absorb should be ideal for this, and I believe I've used it successfully for this purpose in the past. In my most recent instance, though, git-absorb frustratingly refused to do anything in a situation where it felt it should work fine. I had an additional change to a file that was changed in exactly one of my local commits, which feels like an easy case.

(Reading the git-absorb readme carefully suggests that I may be running into a situation where my new change doesn't clash with any existing change. This makes git-absorb more limited than I'd like, but so it goes.)

In Git, what I want is called a 'fixup commit', and how to use it is covered in this Stackoverflow answer. The sequence of commands is basically:

# modify some/file with new changes, then
git add some/file

# Use this to find your existing commit ID
git log some/file

# with the existing commid ID
git commit --fixup=<commit ID>
git rebase --interactive --autosquash <commit ID>^

This will open an editor buffer with what 'git rebase' is about to do, which I can immediately exit out of because the defaults are exactly what I want (assuming I don't want to shuffle around the order of my local commits, which I probably don't, especially as part of a fixup).

I can probably also use 'origin/main' instead of '<commit ID>^', but that will rebase more things than is strictly necessary. And I need the commit ID for the 'git commit --fixup' invocation anyway.

(Sufficiently experienced Git people can probably put together a script that would do this automatically. It would get all of the files staged in the index, find the most recent commit that modified each of them, abort if they're not all the same commit, make a fixup commit to that most recent commit, and then potentially run the 'git rebase' for you.)

How Go behaves if you have a zero value channel or map (a 'nil' channel or map) is somewhat confusing (cf, via). When we talk about it, it's worth remembering that this behavior is a somewhat arbitrary choice on Go's part, not a fundamental set of requirements that stems from, for example, other language semantics. Go has reasons to have channels and maps behave as they do, but some those reasons have to do with how channel and map values are implemented and some are about what's convenient for programming.

As hinted at by how their zero value is called a 'nil' value, channel and map values are both implemented as pointers to runtime data structures. A nil channel or map has no such runtime data structure allocated for it (and the pointer value is nil); these structures are allocated by make(). However, this doesn't entirely allow us to predict what happens when you use nil values of either type. It's not unreasonable for an attempt to assign an element to a nil map to panic, since the nil map has no runtime data structure allocated to hold anything we try to put in it. But you don't have to say that a nil map is empty and looking up elements in it gives you a zero value; I think you could have this panic instead, just as assigning an element does. However, this would probably result in less safe code that paniced more (and probably had more checks for nil maps, too).

Then there's nil channels, which don't behave like nil maps. It would make sense for receiving from a nil channel to yield the zero value, much like looking up an element in a nil map, and for sending to a nil channel to panic, again like assigning to an element in a nil map (although in the channel case it would be because there's no runtime data structure where your goroutine could metaphorically hang its hat waiting for a receiver). Instead Go chooses to make both operations (permanently) block your goroutine, with panicing on send reserved for sending to a non-nil but closed channel.

The current semantics of sending on a closed channel combined with select statements (and to a lesser extent receiving from a closed channel) means that Go needs a channel zero value that is never ready to send or receive. However, I believe that Go could readily make actual sends or receives on nil channels panic without any language problems. As a practical matter, sending or receiving on a nil channel is a bug that will leak your goroutine even if your program doesn't deadlock.

Similarly, Go could choose to allocate an empty map runtime data structure for zero value maps, and then let you assign to elements in the resulting map rather than panicing. If desired, I think you could preserve a distinction between empty maps and nil maps. There would be some drawbacks to this that cut against Go's general philosophy of being relatively explicit about (heap) allocations and you'd want a clever compiler that didn't bother creating those zero value runtime map data structures when they'd just be overwritten by 'make()' or a return value from a function call or the like.

(I can certainly imagine a quite Go like language where maps don't have to be explicitly set up any more than slices do, although you might still use 'make()' if you wanted to provide size hints to the runtime.)

Sidebar: why you need something like nil channels

We all know that sometimes you want to stop sending or receiving on a channel in a select statement. On first impression it looks like closing a channel (instead of setting the channel to nil) could be made to work for this (it doesn't currently). The problem is that closing a channel is a global thing, while you may only want a local effect; you want to remove the channel from your select, but not close down other uses of it by other goroutines.

This need for a local effect pretty much requires a special, distinct channel value that is never ready for sending or receiving, so you can overwrite the old channel value with this special value, which we might as well call a 'nil channel'. Without a channel value that serves this purpose you'd have to complicate select statements with some other way to disable specific channels.

(I had to work this out in my head as part of writing this entry so I might as well write it down for my future self.)

In a comment on my entry on modern languages and bad packaging outcomes at scale, sapphirepaw said (about backward and forward compatibility within language ecologies), well, I'm going to quote from it because it's good (but go read the whole comment):

I think there’s a social contract that has broken down somewhere.

[...]

If a library version did break things, it was generally considered a bug, and developers assumed it would be fixed in short order. Then, for the most part, only distributions had to worry about specific package/library-version incompatibilities.

This all falls apart if a developer, or the ecosystem of libraries/language they depend on, ends up discarding that compatibility-across-time. That was the part that made it feasible to build a distribution from a collection of projects that were, themselves, released across time.

I have a somewhat different view. I think that the way it was in the old days was less a social contract and more an effect of the environment that software was released into and built in, and now that the environment has changed, the effects have too.

C famously has a terrible story around its (lack of a) build system and dependency management, and for much of its life you couldn't assume pervasive and inexpensive Internet connectivity (well, you still can't assume the latter globally, but people have stopped caring about such places). This gave authors of open source software a strong incentive to be both backward and forward compatible. If you released a program that required the features of a very recent version of a library, you reduced your audience to people who already had the recent version (or better) or who were willing to go through the significant manual effort to get and build that version of the library, and then perhaps make all of their other programs work with it, since C environments often more or less forced global installation of libraries. If you were a library author releasing a new minor version or patch level that had incompatibilities, people would be very slow to actually install and adopt that version because of those incompatibilities; most of their programs using your libraries wouldn't update on the spot, and there was no good mechanism to use the old version of the library for some programs.

(Technically you could make this work with static linking, but static linking was out of favour for a long time.)

All of this creates a quite strong practical and social push toward stability. If you wanted your program or its new version to be used widely (and you usually did), it had better work with the old versions of libraries that people already had; requiring new APIs or new library behavior was dangerous. If you wanted the new version of your library to be used widely, it had better be compatible with old programs using the old API, and if you wanted a brand new library to be used by people in programs, it had better demonstrate that it was going to be stable.

Much of this spilled over into other languages like Perl and Python. Although both of these developed central package repositories and dependency management schemes, for a long time these mostly worked globally, just like the C library and header ecology, and so they faced similar pressures. Python only added fully supported virtual environments in 2012, for example (in Python 3.3).

Modern languages like Go and Rust (and the Node.js/NPM ecosystem, and modern Python venv based operation) don't work like that. Modern languages mostly use static linking instead of shared libraries (or the equivalent of static linking for dynamic languages, such as Python venvs), and they have build systems that explicitly support automatically fetching and using specific versions of dependencies (or version ranges; most build systems are optimistic about forward compatibility). This has created an ecology where it's much easier to use a recent version of something than it was in C, and where API changes in dependencies often have much less effect because it's much easier (and sometimes even the default) to build old programs with old dependency versions.

(In some languages this has resulted in a lot of programs and packages implicitly requiring relatively recent versions of their dependencies, even if they don't say so and claim wide backward compatibility. This happens because people would have to take explicit steps to test with their stated minimum version requirements and often people don't, with predictable results. Go is an exception here because of its choice of 'minimum version selection' for dependencies over 'maximum version selection', but even then it's easy to drift into using new language features or new standard library APIs without specifically requiring that version of Go.)

One of the things about technology is that technology absolutely affects social issues, so different technology creates different social expectations. I think that's what's happened with social expectations around modern languages. Because they have standard build systems that make it easy to do it, people feel free to have their programs require specific version ranges of dependencies (modern as well as old), and package authors feel free to break things and then maybe fix them later, because programs can opt in or not and aren't stuck with the package's choices for a particular version. There are still forces pushing towards compatibility, but they're weaker than they used to be and more often violated.

Or to put it another way, there was a social contract of sorts for C libraries in the old days but the social contract was a consequence of the restrictions of the technology. When the technology changed, the 'social contract' also changed, with unfortunate effects at scale, which most developers don't care about (most developers aren't operating at scale, they're scratching their own itch). The new technology and the new social expectations are probably better for the developers of programs, who can now easily use new features of dependencies (or alternately not have to update their code to the latest upstream whims), and for the developers of libraries and packages, who can change things more easily and who generally see their new work being used faster than before.

(In one perspective, the entire 'semantic versioning' movement is a reaction to developers not following the expected compatibility that semver people want. If developers were already doing semver, there would be no need for a movement for it; the semver movement exists precisely because people weren't. We didn't have a 'semver' movement for C libraries in the 1990s because no one needed to ask for it, it simply happened.)

Recently I had to investigate a mysterious issue in our Django based Python web application. This issue happened only when the application was actually running as part of the web server (using mod_wsgi, which effectively runs as an Apache process). The only particularly feasible way to dig into what was going on was everyone's stand-by, print based debugging (because I could print into Apache's error log; I could have used any form of logging that would surface the information). Even if I might have somehow been able to attach a debugger to things to debug a HTTP request in flight, using print based debugging was a lot easier and faster in practice.

I'm a long time fan of print based debugging. Sometimes this is because print based debugging is easier if you only dip into a language every so often, but that points to a deeper issue, which is that almost every environment can print or log. Print or log based 'debugging' is an almost universal way to extract information from a system, and sometimes you have no other practical way to do that.

(The low level programming people sometimes can't even print things out, but there are other very basic ways to communicate things.)

As in my example, one of the general cases where you have very little access other than logs is when your issue only shows up in some sort of isolated or encapsulated environment (a 'production' environment). We have a lot of ways of isolating things these days, things like daemon processes, containers, 'cattle' (virtual) servers, and so on, but they all share the common trait that they deliberately detach themselves away from you. There are good reasons for this (which often can be boiled down to wanting to run in a controlled and repeatable environment), but it has its downsides.

Should print based debugging be the first thing you reach for? Maybe not; some sorts of bugs cause me to reach for a debugger, and in general if you're a regular user of your chosen debugger you can probably get a lot of information with it quite easily, easier than sprinkling print statements all over. But I think that you probably should build up some print debugging capabilities, because sooner or later you'll probably need them.

Recently I read Rust, reflection and access rules. Among other things, it describes how a hypothetical Rust reflection system couldn't safely allow access to private fields of things, and especially how it couldn't allow code to set them through reflection. My short paraphrase of the article's discussion is that in Rust, private fields can be in use as part of invariants that allow unsafe operations to be done safely through suitable public APIs. This brought into clarity what had previously been a somewhat odd seeming restriction in Go's reflect package.

Famously (for people who've dabbled in reflect), you can only set exported struct fields. This is covered in both the Value.CanSet() package documentation and The Laws of Reflection (in passing). Since one of the uses of reflection is for going between JSON and structs, encoding/json only works on exported struct fields and you'll find a lot of such fields in lots of code. This requirement can be a bit annoying. Wouldn't it be nice if you didn't have to make your fields public just to serialize them easily?

(You can use encoding/json and still serialize non-exported struct fields, but you have to write some custom methods instead of just marking struct fields the way you could if they were exported.)

Go has this reflect restriction, presumably, for the same reason that reflection in Rust wouldn't be able to modify private fields. Since private fields in a Go struct may be used by functions and methods in the package to properly manage the struct, modifying those fields yourself is unsafe (in the general sense). The reflect package will let you see the fields (and their values) but not change their values. You're allowed to change exported fields because (in theory) arbitrary Go code can already change the value of those fields, and so code in the struct's package can't count on them having any particular value. It can at least sort of count on private fields having approved values (or the zero value, I believe).

(I understand why the reflect documentation doesn't explain the logic of not being able to modify private fields, since package documentation isn't necessarily the right place for a rationale. Also, perhaps it was considered obvious.)

One of the classical big reason to want union types in Go is so that one can implement the general pattern of an option type, in order to force people to deal explicitly with null values. Except this is not quite true on both sides. The compiler can enforce null value checks before use already, and union and option types by themselves don't fully protect you against null values. Much like people ignore error returns (and the Go compiler allows this), people can skip over that they can't extract an underlying value from their Result value and return a zero value from their 'get a result' function.

My view is that the power of option types is what they do in the rest of the language, but they can only do this if you can express their guarantees in the type system. The important thing you need for this is non-nullable types. This is what lets you guarantee that something is a proper value extracted from an error-free Result or whatever. If you can't express this in your types, everyone has to check, one way or another, or you risk a null sneaking in.

Go doesn't currently have a type concept for 'something that can't be null', or for that matter a concept that is exactly 'null'. The closest Go equivalent is the general idea of zero values, of which nil pointers (and nil interfaces) are a special case (but you can also have zero value maps and channels, which also have special semantics; the zero value of slices is more normal). If you want to make Result and similar types particularly useful in Go, I believe that you need to change this, somehow introducing types that don't have a zero value.

(Such types would likely be a variation of existing types with zero values, and presumably you could only use values or assign to variables of that type if the compiler could prove that what you were using or assigning wasn't a zero value.)

As noted in a comment by loreb on my entry on how union types would be complicated, these 'union' or 'sum' types in Go also run into issues with their zero value, and as Ian Lance Taylor's issue comment says, zero values are built quite deeply into Go. You can define semantics for union types that allow zero values, but I don't think they're really particularly useful for anything except cramming some data structures into a few less bytes in a somewhat opaque way, and I'm not sure that's something Go should be caring about.

Given that zero values are a deep part of Go and the Go developers don't seem particularly interested in trying to change this, I doubt that we're ever going to get the powerful form of union types in Go. If anything like union types appears, it will probably be merely to save memory, and even then union types are complicated in Go's runtime.

Sidebar: the simple zero value allowed union type semantics

If you allow union types to have a zero value, the obvious meaning of a zero value is something that can't have a value of any type successfully extracted from it. If you try the union type equivalent of a type assertion you get a zero value and 'false' for all possible options. Of course this completely gives up on the 'no zero value' type side of things, but at least you have a meaning.

This makes a zero value union very similar to a nil interface, which will also fail all type assertions. At this point my feeling is that Go might as well stick with interfaces and not attempt to provide union types.

Every so often, people wish that Go had enough features to build some equivalent of Rust's Result type or Option type, often so that Go programmers could have more ergonomic error handling. One core requirement for this is what Rust calls an Enum and what is broadly known as a Union type. Unfortunately, doing a real enum or union type in Go is not particularly simple, and it definitely requires significant support by the Go compiler and the runtime.

At one level we easily do something that looks like a Result type in Go, especially now that we have generics. You make a generic struct that has private fields for an error, a value of type T, and a flag that says which is valid, and then give it some methods to set and get values and ask it which it currently contains. If you ask for a sort of value that's not valid, it panics. However, this struct necessarily has space for three fields, where the Rust enums (and generally union types) act more like C unions, only needing space for the largest type possible in them and sometimes a marker of what type is in the union right now.

(The Rust compiler plays all sorts of clever tricks to elide the enum marker if it can store this information in some other way.)

To understand why we need deep compiler and runtime support, let's ask why we can't implement such a union type today using Go's unsafe package to perform suitable manipulation of a suitable memory region. Because it will make the discussion easier, let's say that we're on a 64-bit platform and our made up Result type will contain either an error (which is an interface value) or an int64[2] array. On a 64-bit platform, both of these types occupy 16 bytes, since an interface value is two pointers in a trenchcoat, so it looks like we should be able to use the same suitably-aligned 16-byte memory area for each of them.

However, now imagine that Go is performing garbage collection. How does the Go runtime know whether or not our 16-byte memory area contains two live pointers, which it must follow as part of garbage collection, or two 64-bit integers, which it definitely cannot treat as pointers and follow? If we've implemented our Result type outside of the compiler and runtime, the answer is that garbage collection has no idea which it currently is. In the Go garbage collector, it's not values that have types, but storage locations, and Go doesn't provide an API for changing the type of a storage location.

(Internally the runtime can set and change information about what pieces of memory contain pointers, but this is not exposed to the outside world; it's part of the deep integration of runtime memory allocation and the runtime garbage collector.)

In Go, without support from the runtime and the compiler the best you can do is store an interface value or perhaps an unsafe.Pointer to the actual value involved. However, this probably forces a separate heap allocation for the value, which is less efficient in several ways that the compiler supported version that Rust has. On the positive side, if you store an interface value you don't need to have any marker for what's stored in your Result type, since you can always extract that from the interface with suitable type assertion.

The corollary to all of this is that adding union types to Go as a language feature wouldn't be merely a modest change in the compiler. It would also require a bunch of work in how such types interact with garbage collection, Go's memory allocation systems (which in the normal Go toolchain allocate things with pointers into separate memory arenas than things without them), and likely other places in the runtime.

(I suspect that Go is pretty unlikely to add union types given this, since you can have much of the API that union types present with interface types and generics. And in my view, union types like Result wouldn't be really useful without other changes to Go's type system, although that's another entry.)

PS: Something like this has come up before in generic type sets.

A lot of programs (or applications) that have a 'user interface' mostly don't have a strongly graphical one; instead, they mostly have text, although with special presentation (fonts, colours, underlines, etc) and perhaps controls and meaning attached to interacting with it (including things like buttons that are rendered as text with a border around it). All of these are not just plain text, so programs have to create and manipulate all of them through some API or collection of APIs. Over time, there have sprung up at least two styles of APIs, which I will call external and inline, after how they approach the problem.

The external style API is the older of the two. In the external API, the program makes distinct API calls to do anything other than plain text (well, it makes API calls for plain text, but you have to do something there). If you want to make some text italic or underlined, you have a special API call (or perhaps you modify the context of a 'display this text' API). If you want to attach special actions to things like clicking on a piece of text or hovering the mouse pointer over it, again, more API calls. This leads to programs that make a lot of API calls in their code and are very explicit about what they're doing in their UI. Sometimes this is bundled together with a layout model in the API, where the underlying UI library will flexibly lay out a set of controls so that they accommodate your variously sized and styled text, your buttons, your dividers, and so on.

In the inline style API, you primarily communicate all of this by passing in text that is in some way marked up, instead of plain text that is rendered literally. One form of such inline markup is HTML (and it is popularly styled by CSS). However, there have been other forms, such as XML markup, and even with HTML, you and the UI library will cooperate to attach special meanings and actions to various DOM nodes. Inline style APIs are less efficient at runtime because they have to parse the text you pass in to determine all of this, instead of your program telling the UI library directly through API calls. At the same time, inline style APIs are quite popular at a number of levels. For example, it's popular in UI toolkits to use textual formats to describe your program's UI layout (sometimes this is then compiled into a direct form of UI API calls, and sometimes you hand the textual version to the UI library for it to interpret).

Despite it being potentially less efficient at runtime, my impression is that plenty of programmers prefer the inline style to the external style for text focused applications, where styled text and text based controls are almost all of the UI. My belief is also that an inline style API is probably what's needed for an attractive text focused programming environment.

On the Fediverse, I had a hot take:

Hot take: the enduring popularity of writing applications in a list of environments that starts with Emacs Lisp and goes on to encompass things like Electron shows that we've persistently failed to create a good high level programming system for writing text-focused applications.

(Plan 9's Acme had some good ideas but it never caught on, partly because Plan 9 didn't.)

(By 'text focused' here I mean things that want primarily to display text and have some controls and user interface elements; this is somewhat of a superset of 'TUI' ideas.)

People famously have written a variety of what are effectively applications inside GNU Emacs; there are multiple mail readers, the Magit Git client, at least one news reader, at least one syndication feed reader, and so on. Some of this might be explained by the 'I want to do everything in GNU Emacs' crowd writing things to scratch their itch even if the result is merely functional enough, but several of these applications are best in class, such as Magit (among the best Git clients as far as I know) and MH-E (the best NMH based mail reading environment, although there isn't much competition, and a pretty good Unix mail reading environment in general). Many of these applications could in theory be stand alone programs, but instead they've been written in GNU Emacs Lisp to run inside an editor even if they don't have much to do with Emacs in any regular sense.

(In GNU Emacs, many of these applications extensively rebind regular keys to effectively create their own set of keyboard commands that have nothing to do with how regular Emacs behaves. They sometimes still do take advantage of regular Emacs key bindings for things like making selections, jumping to the start and end of displayed text, or searching.)

A similar thing goes on with Electron-based applications, a fair number of which are fairly text-focused things (especially if you extend text focused things to cover emojis, a certain amount of images, and so on). For a prominent example, VSCode is a GUI text editor and IDE, so much of what it deals with is text, although sometimes somewhat fancied up text (with colours, font choices, various line markings, and so on).

On the Internet, you can find a certain amount of people mocking these applications for the heavy-weight things that they use as host environments. It's my hot take that this is an unproductive and backward view. Programmers don't necessarily like using such big, complex host environments and turn to them by preference; instead, that they turn to them shows that we've collectively failed to create better, more attractive alternatives.

It's possible that this use of heavy weight environments is partly because parts of what modern applications want and need to do are intrinsically complex. For example, a lot of text focused applications want to lay out text in somewhat complex, HTML-like ways and also provide the ability to have interactive controls attached to various text elements. Some of them need to handle and render actual HTML. Using an environment like GNU Emacs or Electron gets you a lot of support for this right away (effectively you get a lot of standard libraries to make use of), and that support is itself complex to implement (so the standard libraries are substantial).

However, I also think we're lacking text focused environments for smaller scale programs, the equivalent of shell scripts or BASIC programs. There have been some past efforts toward things that could be used for this, such as Acme and Tcl/Tk, but they didn't catch on for various reasons.

(At this point I think any viable version of this probably needs to be based around HTML and CSS, although hopefully we don't need a full sized browser rendering engine for it, and I certainly hope we can use a different language than JavaScript. Not necessarily because JavaScript is a bad language or reasonably performing JavaScript engines are themselves big, but partly because using JavaScript raises expectations about the API surface, the performance, the features, and so on, all of which push toward a big environment.)

Recently I wrote about two ways to (maybe) skip 'Dependabot' commits when using git log, and said at the end that I was probably going to set up Git aliases for both approaches. I've now both done that and failed to do that, at the same time. While I have Git aliases for both approaches, the actual git aliases just shell out to shell scripts.

The simpler and more frustrating case is for only seeing authors that aren't Dependabot:

git log --perl-regexp --author='^((?!dependabot\[bot]).*)$'

This looks like it should be straightforward as an alias, but I was unable to get the alias quoting right in my .gitconfig. No matter what I did it either produced syntax errors from Git or didn't work. So I punted by putting the 'git log ...' bit in a shell script (where I can definitely understand the quoting requirements and get them right) and making the actual alias be in the magic git-config format that runs an external program:

[alias]
  ....
  ndlog = !gitndeplog

The reason this case works as a simple alias is that all of the arguments I'd supply (such as a commit range) come after the initial arguments to 'git log'. This isn't the case for the second approach, with attempts to exclude go.mod and go.sum from file paths:

git log -- ':!:go.mod' ':!:go.sum'

The moment I started thinking about how to use this alias, I realized that I'd sometimes want to supply a range of commits (for example, because I just did a 'git pull' and want to see what the newly pulled commits changed). This range has to go in the middle of the command line, which means that a Git alias doesn't really work. And sometimes I might want to supply additional 'git log' switches, like '-p', or maybe supply a file or path (okay, probably I'll never do that). There are probably some sophisticated ways to make this work as an alias, especially if I assume that all of the arguments I supply will go before the '--', but the simple approach was to write a shell script that did the argument handling and invoke it via an alias in the same way as 'git ndlog' does.

Right now the scripts are named in a terse way as if I might want to run them by hand someday, but I should probably rename them both to 'git-<something>'. In practice I'm probably always going to run them as 'git ...', and a git-<something> name makes it clearer what's going on, and easier to find by command completion in my shell if I forget.

I follow a number of projects written in Go that are hosted on Github. Many of these projects enable Github's "Dependabot" feature (also). This use of Dependabot, coupled with the overall Go ecology's habit of relatively frequent small updates to packages, creates a constant stream of Dependabot commits that update the project's go.mod and go.sum files with small version updates of some dependency, sometimes intermixed with people merging those commits (for example, the Cloudflare eBPF Prometheus exporter).

As someone who reads the commit logs of these repositories to stay on top of significant changes, these Dependabot dependency version bumps are uninteresting to me and, like any noise, they make it harder to see what I'm interested in (and more likely that I'll accidentally miss a commit I want to read about that's stuck between two Dependabot updates I'm skipping with my eyes glazed over). What I'd like to be able to do is to exclude these commits from what 'git log' or some equivalent is showing me.

There are two broad approaches. The straightforward and more or less workable approach is to exclude commits from specific authors, as covered in this Stack Overflow question and answer:

git log --perl-regexp --author='^((?!dependabot\[bot]).*)$'

However, this doesn't exclude the commits of people merging these Dependabot commits into the repository, which happens in some (but not all) of the repositories I track. A better approach would be to get 'git log' to ignore all commits that don't change anything other than go.mod and go.sum. I don't think Git can quite do this, at least not without side effects, but we can get close with some pathspecs:

git log -- ':!:go.mod' ':!:go.sum'

(I think this might want to be '!/' for full correctness instead of just '!'.)

For using plain 'git log', this is okay, but it has the side effect that if you use, eg, 'git log -p' to see the changes, any changes a listed commit makes to go.mod or go.sum will be excluded.

The approach of excluding paths can be broadened beyond go.mod and go.sum to include things like commits that update various administrative files, such as things that control various automated continuous integration actions. In repositories with a lot of churn and updates to these, this could be useful; I care even less about a project's use of CI infrastructure than I care about their Dependabot go.mod and go.sum updates.

(I suspect I'll set up Git aliases for both approaches, since they each have their own virtues.)

Over on the Fediverse, I said something:

Bourne shell trivia of the day:
  var=$(program ...)
is the same as
  var="$(program ...)"
so the quotes are unnecessary.

But:
  program2 $(program ...)
is not the same as:
  program2 "$(program ..)"
and often the quotes are vital.

(I have been writing the variable assignment as var="$(...)" for ages without realizing that the quotes were unnecessary.)

This came about because I ran an old shell script through shellcheck, which recommended replacing its use of var=`...` with var=$(...), and then I got to wondering why shellcheck wasn't telling me to write the second as var="$(...)" for safety against multi-word expansions. The answer is of course that multi-word expansion doesn't happen in this context; even if the $(...) produces what would normally be multiple words of output, they're all assigned to 'var' as a single word.

On the one hand, this is what you want; there's almost no circumstance where you want a command that produces multiple words of output to have the first word assigned to 'var' and then the rest interpreted as a command and its arguments. On the other hand, the Bourne shell is generally not known for being friendly about its quoting. It would be perfectly in character for the Bourne shell to require you to quote the '$(...)' even in variable assignment.

On the one hand, shellcheck doesn't complain about the quoted version and it's consistent with quoting $(...) in other circumstances (when it really does matter). On the other hand, you can easily forget or not know (as I did) that the quoting is unnecessary here, and then you can be alarmed when you see an unquoted 'var=$(...)' in the wild or have it suggested. Since I've mostly written the quoted version, I'll probably continue doing so in my scripts unless I'm dealing with a script that already has some unquoted examples, where I should probably make everything unquoted so that no one reading the script in the future ever thinks there's a difference between the two.

Suppose, not hypothetically, that you have two modules, such as a program and a general module that it uses. Through working on the program, you realize that there are some bugs in the general module, so you fix them and then test them in the program by temporarily using a replace directive, or perhaps a workspace. Eventually you're satisfied with the changes to your module, so you commit them and push the change to the public repository. Now you want to update your program's go.mod to use the module version you've just pushed.

As lots of instructions will tell you, this is straightforward; you want some version of 'go get -u', perhaps 'go get -u .'. However, if you try this immediately, you may discover that Go is not updating the module's version. No matter what you do, not even removing the module from 'go.mod' and then go-get'ing it again, will make Go budge. As far as Go seems to be concerned, your module has not updated and the only available version is the previous one.

(It's possible that 'go get -u <module>@latest' will work here, I didn't think to try it when this happened to me.)

As far as I can tell, what is going on here is the Go module proxy. By default, 'go get' will consult the (public) Go module proxy, and the Go module proxy can have a delay between when you push an update to the public repositories and when the module proxy sees it. I assume that under the hood there's various sorts of rate limiting and other caching, since I expect neither the Go proxy nor the various forges out there want the Go proxy to query forges on every single request just in case an infrequently updated module has been updated this time around.

The blunt hammer way of defeating this is to force 'go get -u' to not use the Go module proxy, with 'GOPROXY=direct go get -u'. This will force Go to directly query the public source and so make it notice your just-pushed update.

PS: If you tagged a new version I believe you can hand edit your go.mod to have the new version. This is more difficult if your module is not officially released, has no version tags, and is using the 'v0.0.0-<git information>' format in go.mod.

PPS: Possibly there is another way you're supposed to do this. If so, it doesn't seem to be well documented.

Recently I was writing some Go code involving goroutines. After I was done, I realized that I had used some new small language features added in Go 1.21 and Go 1.22, without really thinking about it, despite not having paid much attention when the features were added. Specifically, what I used are the new builtins of max() and min(), and 'range over integers' (and also a use of clear(), but only in passing).

Ranging over integers may have sounded a bit silly to me when I first read about it, but it turns out that there is one situation where it's a natural idiom, and that's spawning a certain number of goroutines:

for range min(maxpar, len(args)) {
   wg.Add(1)
   go func() {
     resolver()
     wg.Done()
   }()
}

Before Go 1.21, I would have wound up writing this as:

for i := 0; i < maxpar; i++ {
  [...]
}

I wouldn't have bothered writing and using the function equivalent of min(), because it wouldn't be worth the extra hassle for my small scale usage, so I'd always have started maxpar goroutines even if some of them would wind up doing nothing.

The new max() and min() builtins aren't anything earthshaking, and you could do them as generic functions, but they're a nice little ergonomic improvement in Go. Ranging over integers is something you could always do but it's more compact now and it's nice to directly see what the loop is doing (and also that I'm not actually using the index variable for anything in the loop).

(The clear() builtin is nice, but it also has a good reason for existing. I was only using it on a slice, though, where you can fully duplicate its effects.)

Go doesn't strictly need max(), min(), and range over integers (although the latter is obviously connected to ranging over functions, which is important for putting user container types closer to par with builtin ones). But adding them makes it nicer, and they're small (although growing the language and its builtins does have a quiet cost), and Go has never presented itself as a mathematically minimal language.

(Go will have to draw the line somewhere, because there are a lot of little conveniences that could be added to the language. But the Go team is generally conservative and they're broadly in a position to not do things, so I expect it to be okay.)

Over on the Fediverse, I had a belated realization:

This is my face when I realize I have a situation that 'promises'/asynchronously waitable objects would be great for, but I would have to build them by hand in Go. Oh well.

(I want asynchronous execution but to report the results in order, as each becomes available. With promises as I understand them, generate all the promises in an array, wait for each one in order, report results from it, done.)

A common pattern with work(er) pools in Go and elsewhere is that you want to submit requests to a pool of asynchronous workers and you're happy to handle the completion of that work in any order. This is easily handled in Go with a pair of channels, one for requests and the other for completions. However, this time around I wanted asynchronous requests but to be able to report on completed work in order.

(The specific context is that I've got a little Go program to do IP to name DNS lookups (it's in Go for reasons), and on the one hand it would be handy to do several DNS lookups in parallel because sometimes they take a while, but on the other hand I want to print the results in command line order because otherwise it gets confusing.)

In an environment with 'promises' or some equivalent, asynchronous work with ordered reporting of completion is relatively straightforward. You submit all the work and get an ordered collection of Promises or the equivalent, and then you go through in order harvesting results from each Promise in turn. In Go, I think there are two plausible alternatives; you can use a single common channel for results but put ordering information in them, or you can use a separate reply channel for each request. Having done scratch implementations of both, my conclusion is that the separate reply channel version is simpler for me (and in the future I'm not going to be scared off by thoughts of how many channels it can create).

For the common reply channel version, your requests must include a sequence number and then the replies from the workers will also include that sequence number. You'll receive the replies in some random sequence and then it's on you to reassemble them into order. If you want to start processing replies in order before everything has completed, you have to do additional work (you may want, for example, a container/heap).

For the separate reply channel version, you'll be creating a lot of channels (one per request) and passing them to workers as part of the request; remember to give them a one element buffer size, so that workers never block when they 'complete' each request and send the answer down the request's reply channel. However, handling completed requests in order is simple once you've accumulated a block of them:

var replies []chan ...

for _, req := range worktodo {
  // 'pool' is your worker pool
  replies = append(replies, pool.submit(req))
}

for i := range replies {
  v := <- replies[i]
  // process v
}

If a worker has not yet finished processing request number X when you get to trying to use the reply, you simply block on the channel read. If the worker has already finished, it will have sent the reply into the (buffered, remember) channel and moved on, and the reply is ready for you to pick up immediately.

(In both versions, if you have a lot of things to process, you probably want to handle them in blocks, submitting and then draining N items, repeating until you've handled all items. I think this is probably easier to do in the separate reply channel version, although I haven't implemented it yet.)

'Unmaintained' open source code represents a huge amount of value, value that shouldn't and can't be summarily ignored when considering issues like language backward compatibility. Some of that code is more or less unmaintained, but some of it is maintained by people spending a bit of time working on things to keep projects going. It is perhaps tempting to say that such semi-maintained projects should deal with language updates and so on. I maintain that this is wrong.

These people keeping the lights on in these projects often have limited amounts of time that they either can or will spend on their projects. They don't owe the C standard or anyone else any amount of that time, not even if the C standard people think it should be small and insignificant and easy. Outside backward incompatible changes (in anything) that force these people to spend their limited time keeping up (or force them to spend more time) are at the least kind of rude.

(Such changes are also potentially ineffective or dangerous, in that they push people towards not updating at all and locking themselves to old compilers, old compiler settings, old library and package versions, and so on. Or abandoning the project entirely because it's too much work.)

Of course this applies to more than just backward incompatible language changes; especially it applies to API changes. Both language and API changes force project maintainers into a Red Queen's Race, where their effort doesn't improve their project, it just keeps it working. Does this mean that you can never change languages or APIs in ways that break backward compatibility? Obviously not, but it does mean that you should make sure that the change is worth the cost, and the more used your language or API is, the higher the cost. C is an extremely widely used language, so the cost of any break with backward compatibility in it (including in the C standard library) is quite high.

The corollary of this for maintainers is that if you want your project to not require much of your time, you can't depend on APIs that are prone to backward incompatible changes. Unfortunately this may limit the features you can provide or the languages that you want to use (depending not just on the rate of change in the language itself but also in the libraries that the language will force you to use).

(For example, as a pragmatic thing I would rather write a low maintenance TLS using program in Go than in anything else right now, because the Go TLS package is part of the core Go library and is covered by the Go 1 compatibility guarantee. C and C++ may be pretty stable languages and less likely to change than Go, but OpenSSL's API is not.)

We have a long standing Mercurial 'alias' (it's not an alias in the Git sense) called 'hg sdiff' that provides diffs in non-context form, because for system administrator usage we're often changing things where the context of standard (context) diffs isn't useful and we want the terseness of standard diffs. For a long time we've had a little irritation, where if you changed only one file in a Mercurial repository 'hg sdiff' wouldn't show you the file name, but if you changed several files, 'hg sdiff' would. Today I dug into what was going on and it is more peculiar than I expected.

Mercurial has no native 'hg diff' option to do non-context diffs, so we actually do this through the standard Extdiff extension, which allows you to use external programs to provide diffs. We configure our 'sdiff' custom extdiff command to run 'diff -Nr', which on our Ubuntu machines uses GNU diffutils. However, GNU Diff has no command line option to print filenames. Nor are the file names printed by the code of the Extdiff extension itself when it runs your external diff command.

The clue to what is going on is in the '-r' argument to diff, or alternately this sentence in the Extdiff documentation:

The external diff programs are called with a configurable set of options and two non-option arguments: paths to directories containing snapshots of files to compare, or paths to the files themselves if only one file is modified.

If we run 'hg sdiff' and only one file has been changed in the repository, Extdiff will invoke 'diff -Nr file1.old file1' (to simplify the arguments a bit), and diff itself won't print any filenames. However, if we run 'hg sdiff' and we've changed two or more files, Extdiff will create two directories and invoke 'diff -Nr directory1 directory2', and then since it's running in recursive mode, diff will print the filenames along with the changes in the files.

(I believe things may be slightly more complex if you run Extdiff to compare two revisions, instead of comparing a revision to the working tree, but even then our 'hg sdiff' doesn't print the file name for single-file changes.)

As far as I can tell there's no Extdiff option to always create directories and do recursive diffs, which would do what we want here. Extdiff does have the '--per-file' option to do the reverse and always pass your external program two files. One could write a cover script for Extdiff usage that detects the two-files case and prints filenames appropriately, but we're going to just continue to live with the situation.

(We're not interested in switching to zero-context unified or context diffs, both of which would print the filenames but be more verbose and potentially confusing.)

I recently read An unordered list of things I miss in Go (via). One of those things is 'keyword and default arguments for functions', which in the article was not labeled as requiring a major revision to the language. In one sense this is true, but in another sense it's not, because adding keyword arguments to any compiled language raises ABI questions. This is especially the case in Go, which is generally supposed to be a low-overhead language in terms of how things are stored in memory and passed around in function calls (and Go went through an ABI change not too long ago to enable passing function arguments in registers instead of on the stack).

(I think that keyword arguments with default values don't really raise general API issues, assuming that keyword arguments can only be passed as keywords.)

If Go wants keyword arguments to be useful for small functions that will be called often and should be efficient, such as the article's example of strings.Replace() making the final 'n' argument be a keyword argument with a default value, then calling such a function needs to be roughly as efficient as calling a version without a keyword argument. This implies that such arguments should be passed more or less like non-keyword arguments are today, rather than assembled into a separate mechanism that would necessarily have more overhead; basically you'd treat them as regular arguments that can be specified by name instead of position.

Making default values efficient is tricky and has implications for what sort of default values are allowed. If default values must be compile time constants, every call site can add them in for unspecified keyword arguments. If they can be values only established at runtime (as the initial value of variables can be), then you need some scheme to record these values and then either fetch them at the call sites or pass information on what keyword arguments need default arguments to the function being called. If default values are only determined at the time the function is called, you must do the last, but this will probably be the least efficient option.

All of these choices have implications for ABI stability, which affects what Go shared libraries can be used for and how. For instance, people using shared libraries for Go packages would probably like it if adding a new keyword argument to some function did not break existing compiled code that was calling that function. But it certainly would be simpler if all code had to be compiled together with exactly current information and there was no shared library ABI compatibility of the form that is common for C shared libraries.

(In C, adding an extra argument to a function is an API break, but as mentioned, this isn't necessarily true for adding a keyword argument with a default argument. If it's not an API break, it would be convenient if it's not an ABI break either, but that's challenging, especially for efficient calls even with keyword arguments.)

Generally, it would be best if the (hypothetical) Go language specification for function keyword arguments didn't preclude some of these options, even if the main Go compiler and toolchain was not going to use them. Go already has several implementations, so someday there might be an implementation that values C-like ABI stability. Nor do you want to preclude efficient implementations with no such ABI compatibility, although some of the choices for things like what default argument values are allowed affect that.

(Python doesn't have problems with all of this because it's not trying to be Go's kind of language. Python can define abstract semantics without worrying about efficient implementation or whether some of the language semantics require various inefficiencies. And even then, Python has traps in default argument values, and also.)

Recently I read An unordered list of things I miss in Go (via). One of the things is 'keyword and default arguments for functions', to which I had a reaction:

Hot take: having keyword arguments and optional arguments (and default values) for function calls in your language encourages people to make functions that take too many arguments.

(I can see why Python did it, because on the one hand class/object construction and on the other hand, in a dynamic language it lets you change a function's API without having to hunt down absolutely everyone who calls it and now doesn't have that extra argument. Just make the new argument a keyword one, done.)

Technically you can have keyword arguments without supporting optional arguments or default values, but I don't think very many languages do. The languages I've seen with this generally tend to make keyword arguments optional and then sometimes let you set a default value if no value is supplied (otherwise an unsupplied argument typically gets a zero value specific to its type or the language; for example I believe Emacs Lisp makes them all nil).

I'm sure it's possible to make tasteful, limited use of keyword arguments to good benefit; the article suggests one (in Go) and I'm sure I've seen examples in Python. But it's far more common to create sprawling APIs with a dozen or more parameters, such as Python's subprocess.run() (or the even larger subprocess.Popen()). I'm personally not a fan of such APIs, although this is one of those taste issues that is hard to quantify.

(The usual excuse is that you don't normally use all that many of those keyword arguments. But if you do, things are messy, and the API is messy because all of them exist. There are other patterns that can be used for APIs that intrinsically have lots of options; some are common Go practice. These patterns are more verbose, but in my view that verbosity is not necessarily a bad thing.)

Another unfortunate aspect of optional keyword arguments with default values is that they enable a lazy way of expanding and changing function APIs (and method APIs and constructor APIs and so on). Rather than add or change a regular argument and have to update all of the call sites, you add a new keyword argument with a default value, and then only update or add call sites that need to use the new part of the API. I've done this myself because it was quick and easy, and I've also wound up with the end state of this that you sometimes get, where all of my call sites were using the new API with the new keyword argument, so I could have gotten rid of it as a keyword and made it a regular argument.

I also feel that keyword arguments encourage a certain bad way of evolving APIs (by making this way the easiest way to move forward). In one version, some bad behavior is allowed to linger because everyone is supposed to know to turn it off with a keyword argument. In another version, some incompatibility is eventually allowed to be added because everyone who doesn't want it is supposed to have used a keyword argument to disable it. In either situation, if you don't know about the magic trick with the keyword argument, you lose out.

What you can say for keyword arguments is that taking a lot of keyword arguments is better than taking a lot of non-keyword arguments, but to me this is not much of an endorsement. It's better not to have lots of arguments in general.

Crowdstrike recently released an "External Technical Root Cause Analysis" [PDF] (via) for their recent extreme failure. The writeup is rather unclear about what exactly happened, but I think I understand it and if I do, it's an uncomfortably easy programming mistake to make. So here is my version of the core programming issue.

Part of Crowdstrike's agent is a signature matching system, where they match signature patterns (templates) against what's going on. There are different types of patterns (template types) depending on what sort of activity the signature pattern is matching. Rather than have one big regular expression or equivalent in each pattern that matches against all data for the activity, all smashed together and encoded somehow, Crowdstrike's code breaks this out so pattern types have some number of fields (each of them a separate regular expression or equivalent) and are supplied with matching input values (or input data) extracted from the activity in order to do the field by field matching.

(You could invert this so that during the matching of each pattern (template), the matching code calls back to obtain the input values from the activity, but since you can have a lot of different templates for a given template type, it's more efficient to extract the data once and then reuse it across all of the templates of that type. You can still do this with the call back pattern, but it's more complicated.)

For whatever reason, the input values were passed not as separate arguments but as a single variable size array, the input data array (although this might not have been as a literal array but instead as varargs function arguments). It's possible that the core matching code was in fact generic; it was given a template with some number of fields and an input data array with some number of fields and it walked through matching them up. If all of the field matching genuinely is text regular expression matches, this wouldn't be a crazy way to structure the code. You'd have one general matching system plus a collection of template type specific code to that was passed information about the activity and had the job of pulling data out of it and populating the input data array for its patterns.

The mismatch occurred when the IPC templates specified and matched against 21 fields, but the input data array only had 20 pieces of data. The issue had previously been masked because all previous IPC templates had a wildcard match for the 21st field and this skipped the actual comparison. In hand-waved Go code, one version of this might look like:

func (t *Template) DoesMatch(eventdata SomeType[]) bool {
  for i := range t.Fields {
    if t.Fields[i] != MatchesAll &&
       !FieldMatches(t.Fields[i], eventdata[i]) {
      return false
    }
  }
  return true
}

(To be clear, I'm not claiming that this is what Crowdstrike's code was doing. All we know about the actual code from Crowdstrike's report is that it involved an arity mismatch that wasn't detected at compile time.)

Now, you would certainly like this code to start with a check that 'len(t.Fields) == len(eventdata)' so that it doesn't panic if there's a length mismatch. But you might not put that check in, depending on the surrounding context. And in general this code and design is, in my opinion, not particularly bad or unnatural; you might write something like it in all sorts of situations where you have a variable amount of data that needs to be provided to something (or various different things).

Compilers and language environments are broadly not all that good at statically detecting dynamic arity mismatches; it's a hard problem involving things like dataflow analysis. You can sometimes change the code to avoid being dynamic this way, but it might involve awkward restructuring that leads to other issues. For example, you might have a per template type 'myEventData' structure with named fields, insuring that you can never access an undefined field, but then you probably have to have per template type matching code so that you can create and fill in the right myEventData structure and call everything in a type-safe way (although remember to write tests that verify that every field in the structure is actually filled in). This would involve duplicating the logic of 'check all of these templates of the give type' across these functions, and that code might have complexities of its own.

(Modern language environments have done a lot of work to detect arity problems in things like printf() or its equivalents, but this is generally done by specific knowledge of how to determine the arity (and types) given a format string or the like.)

If you have a master source of truth for what the arity should be (how many fields or arguments are required), such as a data file definition, you can build additional tooling to check that the arity is correct or to automatically generate test cases that will verify that. But this requires having that source of truth and building the generally custom tooling to use it. You also have to insure that the test cases are doing what you think they're doing, which might have been another issue in play here (the people writing the IPC template type code and its tests may not have known that wildcard field matches were short circuited so that they didn't even look at the matching element of the input data array).

(IDLs also don't have a great reputation among programmers, partly because there have been a lot of bad IDLs over the years that people have been forced to deal with.)

There are also intermediate positions between the fully dynamic arity issue that Crowdstrike apparently had and the fully type safe version you could write with enough work. But they all involve some runtime dynamic behavior, which means at least a possibility for runtime mistakes, which would have to be carefully caught lest they crash your code. For instance, in Go you could have a per template type structure, pass it through the common code as an interface{}, and then type-assert it back to the real structure type in your per template type matcher function. But you'd have to handle the possibility of that type assertion failing, however unlikely it should be.

PS: None of this should be taken as excusing Crowdstrike. They were writing software that ran in an extremely privileged and important situation, and they should have done that (much) better.

PPS: Neither Go nor Rust by themselves will save you from this specific sort of dynamic arity mistake; they merely change how your code will fail, from an invalid memory dereference crash to a panic in the language runtime or your (macro-expanded) code.

When I wrote about how I felt GNU Emacs configuration bankruptcy was inevitable in the longer term, one of the reasons was that Emacs configuration settings have changed over time (and will change more in the future), so that your old way of configuring something (like the C indentation style) would have to be updated periodically. One reaction to this is to say that open source software should keep backward compatibility for such settings. Unfortunately and as usual, such backward compatibility would have real costs; it would effectively act as a tax on development.

If you promise backward compatibility for settings, you must devote (programming) effort to mapping old settings to the new behavior and perhaps the new settings. Where there's no exact equivalent of the old setting's behavior, you may have to add code or additional (new) settings to synthesize that behavior, or materialize it if someone ever asks for it. Or you can only imitate the old setting value imperfectly, but then (some) people will complain. All of this takes work, especially if the setting is controlling some old behavior and old code that you're trying to move away from.

Open source software has finite time to spend on development (and increasing usage doesn't necessarily scale up the time available the way it can for commercial software). So the more backward compatibility you maintain, the more of your development time goes to that, and the less you're moving forward. And all of this is for something that you certainly hope fewer and fewer people are using over time, with new users and some number of old people moving to your current system of settings.

It's not surprising when large software like GNU Emacs doesn't preserve backward compatibility in settings, especially over the long term. In fact, I'd go further than that; it's a good thing when open source software doesn't attempt to do this if it's at all difficult. I'd much rather see limited resources going on improving the project and moving it forward rather than letting me not modify my .emacs for another year or three.

(Much of this applies to backward compatibility in general, at least in straightforward software that people use directly. Operating systems, libraries, and similar infrastructure things are somewhat different for reasons beyond the scope of this entry.)

Over on the Fediverse, there was a discussion of doing software development things using email and I said something:

My heretical opinion is that I would rather file a Github issue against your project than send you or your bug tracker email, because I do not trust you to safeguard my email against spammers, so I have to make up an entire new email address for you and carefully manage it. I don't trust Github either, but I have already done all of this email address handling for them.

(I also make up an email address for my Git commits. And yes, spammers have scraped it and spammed me.)

Github is merely a convenient example (and the most common one I deal with). What matters is that the forge is a point of centralization (so it covers a lot of projects) and that it does not require me to expose my email to lots of people. Any widely used forge-style environment has the same appeal (and conversely, small scale forges do not; if I am going to report issues to only one project per forge, it is not much different than a per-project bug tracker or bug mailing list).

That email is so much of a hassle today is a bad and sad thing. Email is a widely implemented open standard with a huge suite of tools that allows for a wide range of ways of working with it. It should be a great light-weight way of sending in issues, bug reports, patches, etc etc, and any centralized, non-email place to do this (like Github) has a collection of potential problems that should make open source/free software people nervous.

Unfortunately email has been overrun by spammers in a way that forges have not (yet) been, and in the case of email the problem is essentially intractable. Even my relatively hard to obtain Github-specific email address gets spam email, and my Git commit email address gets more. And demonstrating the problem with not using forges, the email address I used briefly to make some GNU Emacs bug reports about MH-E got spam almost immediately, which shows why I really don't want to have to send my issues by email to an exposed mailing list with public archives.

While there are things that might make the email situation somewhat better (primarily by hiding your email address from as many parties as possible), I don't think there's any general fix for the situation. Thanks to spam and abuse, we're stuck with a situation where setting yourself up on a few central development sites with good practices about handling your contact methods is generally more convenient than an open protocol, especially for people who don't do this all the time.

Recently I read Avoiding Emacs bankruptcy, with good financial habits (via). To badly summarize the article, it suggests avoiding third party packages and minimizing the amount of customization you do. As it happens, I have experience with more or less this approach, and in the end it didn't help. Because I built my old Emacs environment in the days before third party Emacs package management, it didn't include third party packages (although it may have had a few functions I'd gotten from other people). And by modern standards it wasn't all that customized, because I didn't go wildly rebinding keys or the like. Instead, I mostly did basic things like set indentation styles. But over the time from Emacs 18 to 2012, even that stuff stopped working. The whole experience has left me feeling that Emacs bankruptcy is inevitable over the longer term.

The elements pushing towards Emacs bankruptcy are relatively straightforward. First, Emacs wants personal customization in practice, so you will build up a .emacs for your current version of Emacs even if you don't use third party packages. Second, Emacs itself changes over time, or if you prefer the standard, built-in packages change over time to do things like handle indentation and mail reading better. This means that your customizations of them will need updating periodically. Third, the Emacs community changes over time in terms of what people support, talk about, recommend, and so on. If you use the community at all for help, guidance, and the like, what it will be able to help you with and what it will suggest will change over time, and thus so will what you want in your Emacs environment to go with it. Finally, both your options for third party packages and the third party packages themselves will change over time, again forcing you to make changes in your Emacs environment to compensate.

In addition, as the article implicitly admits, that a package is in the Emacs standard library doesn't mean that it can't have problems or effectively be abandoned with little or no changes and updates (for example, the state of Flymake for a long time). Sticking to the packages that come with Emacs can be limiting and restrictive, much like not customizing Emacs at all and accepting all of its defaults. You can work with Emacs that way (and people used to, back in the days before there was a vibrant ecology of third party packages), but you're being potentially hard on yourself in order to reduce the magnitude of something that's probably going to happen to you anyway.

(For instance, until recently not using third party packages would have meant that you did not have support for language servers.)

My view is that in practice, there's no way to leave your Emacs setup alone for a long time. You can go 'bankrupt' in small pieces of work every so often, or in big bangs like the one I went through (although the small pieces approach is more likely if you keep using Emacs regularly).

I don't think this is a bad thing. It's ultimately a choice on the spectrum between evolution and backward compatibility, where both GNU Emacs and the third party ecosystem would rather move forward (hopefully for the better) instead of freezing things once something is implemented.

Recently I read Avoiding Emacs bankruptcy, with good financial habits (via), which sparked some thoughts. One of them is that I feel that GNU Emacs is an editor that winds up with personal customizations from people who use it, even if you don't opt to install any third party packages and stick purely with what comes with Emacs.

There are editors that you can happily use in their stock or almost stock configuration; this is most of how I use vim. In theory you can use Emacs this way too. In practice I think that GNU Emacs is not such an editor. You can use GNU Emacs without any customization and it will edit text and do a variety of useful things for you, but I believe you're going to run into a variety of limitations with the result that will push you towards at least basic customization of built in settings.

I believe that there are multiple issues, at least:

• The outside world can have multiple options where you have to configure the choice (such as what C indentation style to use) that matches your local environment.

• Emacs (and its built in packages) are opinionated and those opinions are not necessarily yours. If opinions clash enough, you'll very much want to change some settings to your opinions.

  (This drove a lot of my customization of GNU Emacs' MH-E mode, although some of that was that I was already a user of (N)MH.)

• You want to (automatically) enable certain things that aren't on by default, such as specific minor modes or specific completion styles. Sure, you can turn on appealing minor modes by hand, but this gets old pretty fast.

• Some things may need configuration and have no defaults that Emacs can provide, so either you put in your specific information or you don't get that particular (built in) package working.

Avoiding all of these means using GNU Emacs in a constrained way, settling for basic Emacs style text editing instead of the intelligent environment that GNU Emacs can be. Or to put it another way, Emacs makes it appealing to tap into its power with only a few minor settings through the built in customization system (at least initially).

I believe that most people who pick GNU Emacs and stick with it want to use something like its full power and capability; they aren't picking it up as a basic text editor. Even without third party packages, this leads them to non-trivial customizations to their specific environment, opinions, and necessary choices.

(Perhaps this is unsurprising and is widely accepted within the GNU Emacs community. Or perhaps there is a significant sub-community that does use GNU Emacs only in its role as a basic text editor, without the various superintelligence that it's capable of.)

Go is working on allowing people to range-over function iterators, and currently this is scheduled to be in Go 1.23, due out this summer (see issue 61405 and issue 61897). The actual implementation is somewhat baroque and some people have been unhappy about that (for example). My view is that this is about bringing user-written container types closer to parity with the special language container types, but recently another view of this occurred to me.

As people have noted, what is most special about this proposal is not that it creates an officially supported iteration protocol in Go, but that this protocol gets direct language support. The compiler itself will transform 'for ... = range afunction' into different code that actively implements the iteration protocol that the Go developers have chosen. This direct language support is critical to making ranging over functions like 'for k,v := range map', but it also does another thing, which is that it implements all of the details of the iteration protocol for the person writing the 'for' loop.

(People seem to generally envision that the actual usage will be 'for ... = range generator(...)', where 'generator()' is a function that returns the actual function that is used for iteration. But I think you could use method values in some situations.)

Iteration protocols are generally fairly complicated. They have to deal with setup, finalization, early exits from process of iteration, finalization in the face of early exits, and so on. The actual implementations of these protocols tends to be gnarly and somewhat subtle, with various potential mistakes and omissions that can be made, and some of these will not manifest in clear bugs until some special situation arises. Go could make everyone who wanted to use 'iterate over a function or special data structure' write out the explicit code needed to do this using the protocol, but if it did we know what the result would be; some of that code would be buggy and incomplete.

By embedding its chosen iteration protocol into the language itself, Go insures that most of that code won't have to be written by you (or by any of the plenty of people who might use user-written types and want to iterate over them). The compiler itself will take a straightforward 'for ... range' block and transform it to correctly and completely implement the protocol. In fact, the protocol is not even particularly accessible to you within the 'for' block you're writing.

People writing the iterator functions for their user-written types will have to care about the protocol, of course (although the Go protocol seems relatively simple in that regard too). But there are likely to be many fewer such iterator creators than there will be iterator users, much as Go assumes that there will be many more people using generic types than people creating them.

When I wrote about the long-overdue problem people are going to have with go:linkname in Go 1.23, the specific case that caused me to notice this was something trying to access crypto/tls's 'defaultCipherSuitesTLS13' variable. As its name suggests, this variable holds the default cipher suites used by Go for TLS 1.3. One reaction to this specific problem is to ask why Go doesn't expose this information as part of crypto/tls's API.

One reason why not is contained in the documentation for crypto/tls.CipherSuites():

[...] Note that the default cipher suites selected by this package might depend on logic that can't be captured by a static list, and might not match those returned by this function.

In fact the TLS 1.3 cipher suites that Go uses may not match the ones in defaultCipherSuitesTLS13, because there is actually a second set of them, defaultCipherSuitesTLS13NoAES. As its name suggests, this set of cipher suites applies when the current machine doesn't have hardware support for AES GCM, or at least hardware support that Go recognizes. Well, even that is too simple a description; if Go is being used as a TLS server, whether the 'no AES GCM' version is used also depends on if the client connecting to the Go server appears to prefer AES GCM (likely signaling that the client has hardware support for it).

Today, Go can't expose a useful API for 'the default TLS 1.3 cipher suites' because there is no such straightforward thing; the actual default cipher suites used depend on multiple factors, some of which can't be used by even a top level function like CipherSuites(). If Go had exported such a variable or API in the past, Go's general attitude on backward compatibility might have forced it to freeze the logic of TLS 1.3 cipher suite choice so that it did respect this default list no matter what, much like the random number generation algorithm became frozen because people depended on it.

The Go 1 compatibility promise is a powerful enabler for Go. But because it is so strong and the Go developers interpret it broadly, it means that Go has to be really careful and certain about what APIs it exposes. As we've seen with math/rand, early decisions to expose certain things, even implicitly, can later constrain Go's ability to make important changes.

PS: Another reason to not expose this information from crypto/tls is that Go has explicitly decided to not make TLS 1.3 cipher suites something that you can control. As covered in the documentation for crypto/tls.Config, the 'CipherSuites' struct element is ignored for TLS 1.3; TLS 1.3 cipher suites are not configurable.

Right now, if you try to build anything using the currently released version of github.com/quic-go/quic-go with the current development version of Go (such as this DNS query program), you will probably encounter the following error:

link: github.com/quic-go/quic-go/internal/qtls: invalid reference to crypto/tls.defaultCipherSuitesTLS13

Experienced Go developers may now be scratching their heads about how quic-go/internal/qtls is referring to crypto/tls.defaultCipherSuitesTLS13, since the latter isn't an exported identifier (in Go, all exported identifiers start with a capital letter). The simple answer is that the qtls package is cheating (in cipher_suite.go).

The official Go compiler has a number of special compiler directives. A few of them are widely known and used, for example '//go:noinline' is common in some benchmarking to stop the compiler from optimizing your test functions too much. One of the not well known ones is '//go:linkname', and for this I'll just quote from its documentation:

//go:linkname localname [importpath.name]

[...] This directive determines the object-file symbol used for a Go var or func declaration, allowing two Go symbols to alias the same object-file symbol, thereby enabling one package to access a symbol in another package even when this would violate the usual encapsulation of unexported declarations, or even type safety. For that reason, it is only enabled in files that have imported "unsafe".

Let me translate that: go:linkname allows you to access unexported variables and functions of other packages. In particular, it allows you to access unexported variables (and functions) from the Go standard library, such as crypto/tls.defaultCipherSuitesTLS13.

The Go standard library uses go:linkname internally to access various unexported things from other packages and from the core runtime, which is perfectly fair; the entire standard library is developed by the same people, and they have to be very careful and conservative with the public API. However, go:linkname has also been used by a wide assortment of third party packages to access unexported pieces of the standard library that those packages found convenient or useful (such as Go's default cipher suites for TLS 1.3). Accessing unexported things from the Go standard library isn't covered by the Go 1 compatibility guarantee, for obvious reasons, but in practice the Go developers find themselves not wanting to break too much of the Go package ecosystem even if said ecosystem is doing unsupported things.

Last week, the Go developers noticed this (I believe not for the first time) and Russ Cox filed issue #67401: cmd/link: lock down future uses of linkname, where you can find a thorough discussion of the issue. The end result is that the current development version of Go, which will become Go 1.23, is now much more restrictive about go:linkname, requiring that the target symbol opt in to this usage. Starting from Go 1.23, you will not be able to 'go:linkname' to things in the standard library that have not specifically allowed this (and the rules are probably going to get stricter in future Go versions; in a few versions I wouldn't be surprised if you couldn't go:linkname into the standard library at all from outside packages).

So this is what is happening with github.com/quic-go/quic-go. It is internally using a go:linkname to get access to crypto/tls's defaultCipherSuitesTLS13, but in Go 1.23, defaultCipherSuitesTLS13 is not one of the symbols that has opted in to this use, so the build is now failing. The quic-go package is probably far from the only package that is going to get caught out by this, now and in the future.

(The Go developers have been adding specific opt-ins for sufficiently used internal identifiers, in files generally called 'badlinkname.go' in the packages. You can see the current state for crypto/tls in its badlinkname.go file.)

As people generally tell the story today, Go was originally developed without support for dependency management. Various community efforts evolved over time and then were swept away in 2019 by Go Modules, which finally added core support for dependency management. I happen to feel that this story is a little bit incomplete and sells the original Go developers short, because I think they did originally have a story for how Go development and dependency management was supposed to work. To me, one of the fascinating bits in Go's evolution to modules is how that original story didn't work out. Today I'm going to outline how I see that original story.

In Go 1.0, the idea was that you would have one or more of what are today called multi-module workspaces. Each workspace contained one (or several) of your projects and all of its dependencies, in the form of cloned and checked-out repositories. With separate repositories, each workspace could have different (and independent) versions of the same packages if you needed that, and updating the version of one dependency in one workspace wouldn't update any other workspace. Your current workspace would be chosen by setting and changing $GOPATH, and the workspace would contain not just the source code but also precompiled build artifacts, built binaries, and so on, all hermetically confined under its $GOPATH.

This story of multiple $GOPATH workspaces allows each separate package or package set of yours to be wrapped up in a directory hierarchy that effectively has all of its dependencies 'vendored' into it. If you want to preserve this for posterity or give someone else a copy of it, you can archive or send the whole directory tree, or at least the src/ portion of it. The whole thing is fairly similar to a materialized Python virtual environment.

(The original version of Go did not default $GOPATH to $HOME/go, per for example the Go 1.1 release notes. It would take until Go 1.8 for this default to be added.)

This story broadly assumes that updates to dependencies will normally be compatible, because otherwise you really want to track the working dependency versions even in a workspace. While you can try to update a dependency and then roll it back (since you normally have its checked out repository with full history), Go won't help you by remembering the identity of the old, working version. It's up to you to dig this out with tools like the git reflog or your own memory that you were at version 'x.y.z' of the package before you updated it. And 'go get -u' to update all your dependencies at once only makes sense if their new versions will normally all work.

This story also leaves copying workspaces to give them to someone else (or to preserve them in their current state) as a problem for you, not Go. However, Go did add 'experimental' support for vendoring dependencies in Go 1.5, which allowed people to create self-contained objects that could be used with 'go get' or other simple repository copying and cloning. A package that had its dependencies fully vendored was effectively a miniature workspace, but this approach had some drawbacks of its own.

I feel this original story, while limited, is broadly not unreasonable. It could have worked, at least in theory, in a world where preserving API compatibility (in a broad sense) is much more common than it clearly is (or isn't) in this one.

When I wrote about understanding the orderless package, I mentioned that orderless doesn't work well with hierarchical completions such as file names, which are completed one component at a time. I also said this mattered to me because MH-E completed the names of mail folders in this part by part manner, but I didn't feel like rewriting MH-E's folder completion system to fix it. Well, you can probably guess what happened next.

In the GNU Emacs way, I didn't so much rewrite MH-E's mail folder completion as add a second folder completion system along side it, and then rebound some keys to use my system. Writing my system was possible because it turned out MH-E had already done most of the work for me, by being able to collect a complete list of all folder names (which it used to support its use of the GNU Emacs Speedbar).

To put the summary up front, I was pleasantly surprised by how easy it was to add my own completion stuff and make use of it within my MH-E environment. At the same time, reverse engineering some of MH-E's internal data structures was a bit annoying and it definitely feels like a bit of a hack (although one that's unlikely to bite me; MH-E is not exactly undergoing rapid and dramatic evolution these days, so those data structures are unlikely to change).

There are many sophisticated way to do minibuffer completion in GNU Emacs, but if your purpose is to work well with orderless, the simplest approach is to generate a list of all of your completion candidates up front and then provide this list to completing-read. This results in code that looks like this:

(defvar cks/mh-folder-history '() "History of MH folder targets.")
(defun cks/mh-get-folder (msg)
  (let ((cks/completion-category 'mh-e-folder-full))
    (completing-read msg (cks/mh-all-folders) nil t "+" cks/mh-folder-history)))

Here I've made the decision that this completion interface should require that I select an existing MH mail folder, to avoid problems. If I want to create a new mail folder I fall back to the standard MH-E functions, with their less convenient completion but greater freedom. I've also decided to give this completion a history, so I can easily re-use my recent folder destinations.

(The cks/completion-category stuff is for forcing the minibuffer completion category so that I can customize how vertico presents it, including listing those recent folder destinations first.)

This 'get MH folder' function is then used in a straightforward way:

(defun mh-refile-msg-full (range folder)
  (interactive (list (mh-interactive-range "Refile")
                     (intern (cks/mh-get-folder "Refile to folder? "))))
  (mh-refile-msg range folder))

This defers all of the hard work to the underlying MH-E command for refiling messages. This is one of the great neat tricks in GNU Emacs with the (interactive ...) form; when you make a function a command with (interactive ...), it's natural to find up with it callable from other ELisp code with the arguments you'd normally be prompted for interactively. So I can reuse the mh-refile-msg command non-interactively, sticking my own interactive frontend on it.

All of the hard work happens in cks/mh-all-folders. Naturally, MH-E maintains its own data structures in a way that it finds convenient, so its 'mh-sub-folders-cache' hash table is not structured as a list of all MH folder names but instead has hash entries storing all of the immediate child folders of a parent plus some information on each (at the root, the 'parent' is nil). So we start with a function to transform various combination of a hash key and a hash value into a MH folder name:

(defun cks/mh-hash-folder-name (key elem)
  (cond
   ((and key elem) (concat key "/" (car elem)))
   (key key)
   (elem (concat "+" (car elem)))))

And then we go over mh-sub-folders-cache using our mapping function with:

(cl-loop for key being the hash-keys of mh-sub-folders-cache
  using (hash-values v)
  collect (cks/mh-hash-folder-name key nil)
  append (cl-loop for sub in v
		  collect (cks/mh-hash-folder-name key sub)))))

After getting this list we need to sort it alphabetically, and also remove duplicate entries just in case (and also a surplus nil entry), using the following:

(sort (remq nil (seq-uniq flist)) 'string-lessp)

(Here, 'flist' is the let variable I have stuck the cl-loop result into. My actual code then removes some folder names I don't want to be there cluttering up the completion list for various reasons.)

There are some additional complications because MH-E will invalidate bits of its sub-folders cache every so often, so we may need to force the entire cache to be rebuilt from scratch (which requires some hackery, but turns out to be very fast these days). I'm not putting those relatively terrible hacks down here (also, the whole thing is somewhat long).

(If I was a clever person I would split this into two functions, one of which generated the full MH mail folder list and the second of which filtered out the stuff I don't want in it. Then I could publish the first function for people's convenience, assuming that anyone was interested. However, my ELisp often evolves organically as I realize what I want.)

My major use of Git is to keep copies of the public repositories of various projects from various people. Every so often, one of the people involved gets sufficiently irritated with the life of being an open source maintainer and takes their project's repository private (or their current Git 'forge' host does it for other reasons). When this happens, on my next 'git pull', Git skids to a halt with:

; git pull
Username for 'https://gitlab.com':

This is not a useful authentication prompt for me. I have no special access to these repositories; if anonymous access doesn't work, there is nothing I can enter for a username and password that will improve the situation. What I want is for Git to fail with a pull error, the same way it would if the repository URL returned a 404 or the connection to the host timed out.

(Git prompts you here because sometimes people do deal with private repositories which they have special credentials for.)

As far as I know, Git unfortunately has no configuration option or command line option that is equivalent to OpenSSH's 'batch mode' for ssh, where it will never prompt you for password challenges and will instead just fail. The closest you can come is setting core.askPass to something that generates output (such as 'echo'), in which case Git will try to authenticate with that bogus information, fail, and complain much more verbosely, which is not the same thing (among other issues, it causes the Git host to see you as trying invalid login credentials, which may have consequences).

If you're running your 'git pull' invocations from a script, as I often am, you can have the script set 'GIT_TERMINAL_PROMPT=0' (and export it into the environment). According to the documentation, this causes Git to fail rather than prompting you for anything, including authentication. It seems somewhat dangerous to set this generally in my environment, since I have no idea what else Git might someday want to prompt me about (and obviously if you need to sometimes get prompted you can't set this). Apparently this is incomplete if you fetch Git repositories over SSH, but I don't do that for public repositories that I track.

(I found this environment variable along with a lot of other discussion in this serverfault question and its answers.)

Some environments that run git behind the scenes, such as the historical 'go get' behavior, default to disabling git prompts. If you use such an environment it may have already handled this for you.

Often, programs have build time configuration settings for features they include, paths they use, and so on. Some of the time, people suggest that the way to handle these is not through systems like 'configure' scripts (whether produced by Autoconf or some other means) but instead by having people edit their settings into things such as your Makefiles or header files ('source code' in a broad sense). As someone who has spent a bunch of time and effort building other people's software over the years, my strong opinion is that you should not do this.

The core problem of this approach is not that you require people to know the syntax of Makefiles or config.h or whatever in order to configure your software, although that's a problem too. The core problem is you're having people modify files that you will also change, for example when you release a new version of your software that has new options that you want people to be able to change or configure. When that happens, you're making every person who upgrades your software deal with merging their settings into your changes. And merging changes is hard and prone to error, especially if people haven't kept good records of what they changed (which they often won't if your configuration instructions are 'edit these files').

One of the painful lessons about maintaining systems that we've learned over the years is that you really don't want to have two people changing the same file, including the software provider and you. This is the core insight behind extremely valuable modern runtime configuration features such as 'drop-in files' (where you add or change things by putting your own files into some directory, instead of everything trying to update a common file). When you tell people to configure your program by editing a header file or a Makefile or indeed any file that you provide, you're shoving them back into this painful past. Every new release, every update they pull from your VCS, it's all going to be a source of pain for them.

A system where people maintain (or can maintain) their build time configurations entirely outside of anything you ship is far easier for people to manage. It doesn't matter exactly how this is implemented and there are mny options for relatively simple systems; you certainly don't need GNU Autoconf or even CMake.

The corollary to this is that if you absolutely insist on having people configure your software by editing files you ship, those files should be immutable by you. You should ship them in some empty state and promise never to change that, so that people building your software can copy their old versions from their old build of your software into your new release (or never get a merge conflict when they pull from your version control system repository). If your build system can't handle even this restriction, then you need to rethink it.

In the wake of the XZ Utils backdoor, which involved GNU Autoconf, it's been somewhat popular to call for Autoconf to go away. Over on the Fediverse I said something about that:

Hot take: autoconf going away would be a significant net loss to OSS, perhaps as bad as the net loss of the Python 2 to Python 3 transition, and for much the same reason. There are a lot of projects out there that use autoconf/configure today and it works, and they would all have to do a bunch of work to wind up in exactly the same place ('a build system that works and has some switches and we can add our feature checks to').

(The build system can never supply all needed tests. Never.)`

Autoconf can certainly be replaced in general, either by one of the existing and more modern configuration and build systems, such as CMake, or by something new. New projects today often opt for one of the existing alternative build systems and (I believe) often find them simpler. But what can't be replaced easily is autoconf's use in existing projects, especially projects that use autoconf in non-trivial ways.

You can probably convert most projects to alternate build systems. However, much of this work will have to be done by hand, by each project that is converted, and this work (and the time it takes) won't particularly move the project forward. That means you're asking (or demanding) projects to spend their limited time to merely wind up in the same place, with a working build system. Further, some projects will still wind up running a substantial amount of their own shell code as part of the build system in order to determine and do things that are specific to the project.

(Although it may be an extreme example, you can look at the autoconf pieces that OpenZFS has in its config/ subdirectory. Pretty much all of that work would have to be done in any build system that OpenZFS used, and generally it would have to be significantly transformed to fit.)

There likely would be incremental security-related improvements even for such projects. For example, I believe many modern build systems don't expect you to ship their generated files the way that autoconf sort of expects you to ship its generated configure script (and the associated infrastructure), which was one part of what let the XZ backdoor slip files into the generated tarballs that weren't in their repository. But this is not a particularly gigantic improvement, and as mentioned it requires projects to do work to get it, possibly a lot of work.

You also can't simplify autoconf by declaring some standard checks obsolete and dropping everything to do with them. It may indeed be the case that few autoconf based programs today are actually going to cope with, for example, there being no string.h header file (cf), but that doesn't mean you can remove mentioning it from the generated header files and so on, since existing projects require those mentions to work right. The most you could do would be to make the generated 'configure' scripts simply assume a standard list of features and put them in the output those scripts generate.

(Of course it would be nice if projects using autoconf stopped making superstitious use of things like 'HAVE_STRING_H' and just assume that standard headers are present. But projects generally have more important things to spend limited time on than cleaning up header usage.)

PS: There's an entire additional discussion that we could have about whether 'supply chain security' issues such as Autoconf and release tarballs that can't be readily reproduced by third parties are even the project's problem in the first place.

One of the things I've had to wrestle with due to my move to reading my email with MH-E in GNU Emacs is that any number of Emacs modes involved in this like to be helpful by reformatting and annotating your email messages in various ways. Often it's not obvious to an outsider what mode (or code) is involved. For what I believe are historical reasons, a lot of MIME handling code has wound up in GNUS (also), which was originally a news reader; some of the code and variables has 'gnus' prefixes while others has 'mm' or 'mml' prefixes. In MH-E (and I believe most things that use Emacs' standard GNUS-based MIME handling), by default you will get nominally helpful things like message fontisizing and maybe highlighting of certain whitespace that the code thinks you might care about. I mostly don't want this, so I have been turning it off where I saw it and could identify the cause.

(As far as message fontisizing goes, sometimes I don't object to it but I very much object to the default behavior of hiding the characters that triggered the fontisizing. I don't want bits of message text hidden on me so that I have to reverse engineer the actual text from visual appearance changes that I may or may not notice and understand.)

Recently I was reading an email message and there was some white space in it that Emacs had given red underlines, causing me to get a bit irritated. People who are sufficiently familiar with GNU Emacs have already guessed the cause, and in fact the answer was right there in what I saw from Leah Neukirchen's suggestion of looking at (more or less) 'C-u C-x ='. What I was seeing was GNU Emacs' default handling of various special space characters.

(I was going to say that this was a non-breaking space, but it turns out not to be; instead it was U+2002, 'en space'. A true non-breaking space is U+00A0.)

As covered in How Text Is Displayed, Emacs normally displays these special characters and others with the (Emacs) nobreak-space face, which (on suitable displays) renders the character as red with a (red) underline. Since all space variants have nothing to render, you get a red underline. As covered in the documentation, you can turn this off generally or for a buffer by setting nobreak-char-display to nil, which I definitely won't be doing generally but might do for MH-E mail buffers, since my environment generally maps special space characters to a plain space if I paste them into terminals and the like.

(A full list of Emacs font faces is in Standard Faces.)

Zero-width spaces (should I ever encounter any in email or elsewhere) are apparently normally displayed using Glyphless Character Display's 'thin-space' method, along with other glyphless characters, and are Unicode U+200B. It's not clear to me if these will display with a red underline in my environment (see this emacs.stackexchange question and answers). Some testing suggests that zero width spaces may hide out without a visual marker (based on using 'C-x 8 RET' aka 'insert-char' to enter a zero-width space, a key binding which I also found out about through this exercise). At this point I am too lazy to figure out how to force zero-width spaces to be clearly visible.

PS: Other spaces known by insert-char include U+2003 (em space), U+2007 (figure space), U+2005 (four per em space), U+200A (hair space), U+3000 (ideographic space), U+205F (medium mathematical space), U+2008 (punctuation space), U+202F (narrow non-breaking space), and more. It's slightly terrifying. Most of the spaces render in the same way. I probably won't remember any of these Unicode numbers, but maybe I can remember C-u C-x = and that 'nobreak-space' as an Emacs face is an important marker.

PPS: Having gone through all of this, it's somewhat tempting to write some ELisp that will let me flip back and forth between displaying these characters in some clearly visible escaped form and displaying them 'normally' (showing as (marked) spaces and so on). That way I could normally see them very clearly, but make them unobtrusive if I had to deal with something that full of them in a harmless way. This is one of the temptations of GNU Emacs (or in general any highly programmable environment).

Today I discovered a weird limitation of some IP address lookup stuff on the Linux machines I use (a limitation that's apparently not universal). In response to this, I rewrote the little Python program that I had previously been using for looking up IP addresses as a Go program, because I was relatively confident I could get Go to work (although it turns out I couldn't use net.LookupAddr() and had to be slightly more complicated). I could have made the Go program a basically straight port of the Python one, but as I was writing it, I couldn't resist polishing off some of the rough edges and adding missing features (some of which the Python program could have had, and some which would have been awkward to add).

This isn't even the first time this particular program has been polished as part of re-doing it; it was one of the Python programs I added things to when I moved them to Python 3 and the argparse package. That was a lesser thing than the Go port and the polishing changes were smaller, but they were still there.

This 'reimplementation leads to polishing' thing is something I've experienced before. It seems that more often than not, if I'm re-doing something I'm going to make it better (or at least what I consider better), unless I'm specifically implementing something with the goal of being essentially an exact duplicate but in a faster environment (which happened once). It doesn't have to be a reimplementation in a different language, although that certainly helps; I've re-done Python programs and shell scripts and had it lead to polishing.

One trigger for polishing is writing new documentation and code comments. In a pattern that's probably familiar to many programmers, when I find myself about to document some limitation or code issue, I'll frequently get the urge to fix it instead. Or I'll write the documentation about the imperfection, have it quietly nibble at me, and then go back to the code so I can delete that bit of the documentation after all. But some of what drives this polishing is the sheer momentum of having the code open in my editor and already changing or writing it.

Why doesn't happen when I write the program the first time? I think part of it is that I understand the problem and what I want to do better the second time around. When I'm putting together the initial quick utility, I have no experience with it and I don't necessarily know what's missing and what's awkward; I'm sort of building a 'minimum viable product' to deal with my immediate need (such as turning IP addresses into host names with validation of the result). When I come back to re-do or re-implement some or all of the program, I know both the problem and my needs better.

Over on the Fediverse, I had a realization:

This is my face when I realize that on a big multi-core machine, I want to do 'sed ... | sed ... | sed ...' instead of the nominally more efficient 'sed -e ... -e ... -e ...' because sed is single-threaded and if I have several costly patterns, multiple seds will parallelize them across those multiple cores.

Even when doing on the fly shell pipelines, I've tended to reflexively use 'sed -e ... -e ...' when I had multiple separate sed transformations to do, instead of putting each transformation in its own 'sed' command. Similarly I sometimes try to cleverly merge multi-command things into one command, although usually I don't try too hard. In a world where you have enough cores (well, CPUs), this isn't necessarily the right thing to do. Most commands are single threaded and will use only one CPU, but every command in a pipeline can run on a different CPU. So splitting up a single giant 'sed' into several may reduce a single-core bottleneck and speed things up.

(Giving sed multiple expressions is especially single threaded because sed specifically promises that they're processed in order, and sometimes this matters.)

Whether this actually matters may vary a lot. In my case, it only made a trivial difference in the end, partly because only one of my sed patterns was CPU-intensive (but that pattern alone made sed use all the CPU it could get and made it the bottleneck in the entire pipeline). In some cases adding more commands may add more in overhead than it saves from parallelism. There are no universal answers.

One of my lessons learned from this is that if I'm on a machine with plenty of cores and doing a one-time thing, it probably isn't worth my while to carefully optimize how many processes are being run as I evolve the pipeline. I might as well jam more pipeline steps whenever and wherever they're convenient. If it's easy to move one step closer to the goal with one more pipeline step, do it. Even if it doesn't help, it probably won't hurt very much.

Another lesson learned is that I might want to look for single threaded choke points if I've got a long-running shell pipeline. These are generally relatively easy to spot; just run 'top' and look for what's using up all of one CPU (on Linux, this is 100% CPU time). Sometimes this will be as easy to split as 'sed' was, and other times I may need to be more creative (for example, if zcat is hitting CPU limits, maybe pigz can help a bit.

(If I have the fast disk space, possibly un-compressing the files in place in parallel will work. This comes up in system administration work more than you'd think, since we can want to search and process log files and they're often stored compressed.)

Today I learned about the M-X command (well, key binding) (via), which "[queries the] user for a command relevant to the current mode, and then execute it". In other words it's like M-x but it restricts what commands it offers to relevant ones. What is 'relevant' here? To quote the docstring:

[...] This includes commands that have been marked as being specially designed for the current major mode (and enabled minor modes), as well as commands bound in the active local key maps.

If you're someone like me who has written some Lisp commands to customize your experience in a major mode like MH-E, you might wonder how you mark your personal Lisp commands as 'specially designed' for the relevant major mode.

In modern Emacs, the answer is that this is an extended part of '(interactive ...)', the normal Lisp form you use to mark your Lisp functions as commands (things which will be offered in M-x and can be run interactively). As mentioned in the Emacs Lisp manual section Using interactive, 'interactive' takes additional arguments to label what modes your command is 'specially designed' for; more discussion is in Specifying Modes For Commands. The basic usage is, say, '(interactive "P" mh-folder-mode)'

If your commands already take arguments, life is simple and you can just put the modes on the end. But not all commands do (especially for quick little things you do for yourself). If you have just '(interactive)', the correct change is to make it '(interactive nil mh-folder-mode)'; a nil first argument is how you tell interactive that there is no argument.

(Don't make my initial mistake and assume that '(interactive "" mh-folder-mode)' will work. That produced a variety of undesirable results.)

Is it useful to do this, assuming you have personal commands that are truly specific to a given mode (as I do for commands that operate on MH messages and the MH folder display)? My views so far are a decided maybe in my environment.

First, you don't need to do this if your commands have keybindings in your major mode, because M-X (execute-extended-command-for-buffer) will already offer any commands that have keybindings. Second, my assortment of packages already gives me quite a lot of selection power to narrow in on likely commands in plain M-x, provided that I've named them sensibly. The combination of vertico, marginalia, and orderless let me search for commands by substrings, easily see a number of my options, and also see part of their descriptions. So if I know I want something to do with MH forwarding I can type 'M-x mh forw' and get, among other things, my function for forwarding in 'literal plaintext' format.

With that said, adding the mode to '(interactive)' isn't much work and it does sort of add some documentation about your intentions that your future self may find useful. And if you want a more minimal minibuffer completion experience, it may be more useful to have a good way to winnow down the selection. If you use M-X frequently and you have commands you want to be able to select in it in applicable modes without having them bound to keys, you really have no choice.

In Go 1.22, the Go developers have made available a "range over function" experiment, as described in the Go Wiki's "Rangefunc Experiment". Recently I read a criticism of this, Richard Ulmer's Questioning Go's range-over-func Proposal (via). As I read Ulmer's article, it questions the utility of the range over func (proposed) feature based on the grounds that this isn't a significant enough improvement in standard library functions like strings.Split (which is given as an example in the "more motivation" section of the wiki article).

I'm not unsympathetic to this criticism, especially when it concerns standard library functionality. If the Go developers want to extend various parts of the standard library to support streaming their results instead of providing the results all at once, then there may well be better, lower-impact ways of doing so, such as developing a standard API approach or set of approaches for this and then using this to add new APIs. However, I think that extending the standard library into streaming APIs is by far the less important side of the "range over func" proposal (although this is what the "more motivation" section of the wiki article devotes the most space to).

Right from the beginning, one of the criticisms of Go was that it had some privileged, complex builtin types that couldn't be built using normal Go facilities, such as maps. Generics have made it mostly possible to do equivalents of these (generic) types yourself at the language level (although the Go compiler still uniquely privileges maps and other builtin types at the implementation level). However, these complex builtin types still retain some important special privileges in the language, and one of them is that they were the only types that you could write convenient 'range' based for loops.

In Go today you can write, for example, a set type or a key/value type with some complex internal storage implementation and make it work even for user-provided element types (through generics). But people using your new container types cannot write 'for elem := range set' or 'for k, v := range kvstore'. The best you can give them is an explicit push or pull based iterator based on your type (in a push iterator, you provide a callback function that is given each value; in a pull iterator, you repeatedly call some function to obtain the next value). The "range over func" proposal bridges this divide, allowing non-builtin types to be ranged over almost as easily as builtin types. You would be able to write types that let people write 'for elem := range set.Forward()' or 'for k, v := kvstore.Walk()'.

This is an issue that can't really be solved without language support. You could define a standard API for iterators and iteration (and the 'iter' package covered in the wiki article sort of is that), but it would still be more code and somewhat awkward code for people using your types to write. People are significantly attracted to what is easy to program; the more difficult it is to iterate user types compared to builtin types, the less people will do it (and the more they will use builtin types even when they aren't a good fit). If Go wants to put user (generic) types on almost the same level (in the language) as builtin types, then I feel it needs some version of a "range over func" approach.

(Of course, you may feel that Go should not prioritize putting user types on almost the same level as builtin types.)