I use Apple devices very often. Overall, I like macOS. Certainly more than Windows.

One of the things I find extremely useful is a command I discovered not too long ago: pbcopy.

pbcopy can be used to copy to the clipboard whatever it receives from standard input. For example, when I am in a shell, I often use a command like this:

cat filename.md | pbcopy

At that point I know that the content of the file is in the clipboard, and I can paste it wherever I need, calmly and without any additional steps.

There is one limitation, though: this only works locally. It works when I am using my Mac and I want to copy something from the macOS shell.

When I connect to a remote (*BSD, Linux, illumos based) server via ssh, pbcopy is not available. Or, more precisely, even if I create a command with the same name on the server, that command cannot directly talk to the clipboard of my Mac in the usual way.

Luckily, modern terminal emulators have a few tricks available.

I use iTerm2 for most of my ssh sessions and, once I realised how useful it would be to have something similar to pbcopy in remote sessions too, I created a small script that works both on Linux, the BSDs and illumox based OSes.

The caveat is that the remote server cannot "magically" access the clipboard of my Mac. So the trick works because the remote command prints a special terminal escape sequence, and the local terminal emulator interprets it.

The sequence is called OSC 52. In short, it allows a program running inside the terminal to ask the terminal emulator to put some base64-encoded text into the local clipboard. This means that support depends on the terminal emulator I am using locally.

I use iTerm2, which supports OSC 52. Other terminal emulators support it too, so the idea is not tied exclusively to iTerm2. However, Apple's default Terminal.app does not appear to support OSC 52, so I would not expect this specific solution to work there.

So, in practice:

• with iTerm2, it works;
• with other OSC 52-compatible terminals, it should work (I haven't tested it, but should work with kitty, Ghostty, etc.);
• with Apple's Terminal.app, at least at the time of writing, it should not.

Before creating the command on the remote server, a specific iTerm2 option needs to be enabled:

Settings -> General -> Selection -> Applications in terminal may access clipboard

This option allows programs running inside the terminal to access the clipboard through escape sequences.

Of course, this has security implications. A program running in the terminal, including a program running on a remote server through ssh, may be able to write to the local clipboard. For my use case this is acceptable, but it is worth knowing what is happening.

All I need to do is create a command, a small sh script. I log into the server where I want to create the command. In my case, I usually create a file called /usr/local/bin/pbcopy with the following content:

#!/bin/sh
printf '\033]52;c;%s\a' "$(base64 | tr -d '\n')"

Then I make it executable:

chmod a+rx /usr/local/bin/pbcopy

From that moment on, I can use pbcopy on the remote server too, piping another command into it:

cat filename.md | pbcopy

The content will not end up in the clipboard of the remote server. It will end up in the local clipboard of my Mac, because iTerm2 receives the OSC 52 sequence and updates the macOS clipboard.

I have already written about how FediMeteo was born, and about how HAProxy helps reduce the number of requests that reach snac.

Seen from the outside, FediMeteo almost seems still. There is a static homepage, regenerated every hour. There are the city pages, with their forecasts. There are RSS feeds waiting to be fetched, JSON objects waiting to be requested, Fediverse instances refreshing data, subscribing, unsubscribing, retrieving profiles, and reading notes.

That is the visible part.

Behind it, however, FediMeteo is much more than a homepage, a few ActivityPub accounts, and a well-behaved reverse proxy. It is a chain of small pieces, in proper Unix style, each trying to do one thing and do it as well as possible.

That chain, although almost invisible from the outside, was not born already tidy. It changed, was rewritten, adapted to new countries, timezones, ambiguous city names, external service limits, and also to my own mistakes.

Some mistakes were small. Others were much less so.

Because FediMeteo is a human project and, as such, imperfect. Imperfect in the way humans are imperfect, which today almost seems unfashionable. I like that.

The first version of the bot was almost embarrassingly simple, and I was proud of that.

It took a city name as input, asked Nominatim for the coordinates through geopy, called the Open-Meteo API for the current weather and the next several days, and printed a markdown block with current conditions, the forecast for today, the next twelve hours, and the coming days. The text was in Italian. The cities were Italian. The timezone was Europe/Rome. There was nothing to calculate.

Around the script, a small sh wrapper read a list of cities and, for each one, ran the Python program and piped its output into snac note_unlisted. A cron job ran the wrapper every six hours. The output was loose markdown, which snac happily renders, and the integration was: standard output goes into standard input. Nothing fancier than that.

I like this kind of design. It is the part of the Unix philosophy that survives even when fashions change.

When I started adding other European countries, I did not need to change much. I separated the operational logic from the localized strings, moved the strings into one JSON file per country, and spread the cron entries so that not every country posted in the same minute. Each country had its own snac instance, in its own FreeBSD jail, with its own dataset. The bot, internally, was almost the same script as before.

This worked because Europe is, in essence, two or three timezones across most of the countries I cared about.

Then I added Germany, and Germany taught me my first lesson about names.

There are several places called Neustadt in Germany. There is a Frankfurt am Main, and a Frankfurt an der Oder, and they are not the same city. There is a Halle in Saxony-Anhalt and a Halle in North Rhine-Westphalia. Asking Nominatim for "Frankfurt, Germany" produced one of the two, consistently, but not always the one I wanted. Some German users wrote to me, politely, to point out that the forecast for "their" Frankfurt was, in fact, for the other one.

I started thinking about disambiguation, but only enough to fix the immediate cases. The bot still took a single city name. The ambiguous ones I worked around by editing the cities file and hoping for the best.

In hindsight, this was the seed of what would happen later.

The United States broke every assumption the bot had grown up with.

The first problem was the number of cities. I wanted reasonable coverage at state level, which meant identifying the main cities for each of the fifty states. The list ended up at more than 1200 entries. That alone is more cities than every other country in the project combined.

The second problem was timezones. The contiguous United States covers four of them, and Alaska and Hawaii bring the total to six. A "current weather at 12:00" line generated at the same instant for New York and for Los Angeles is technically the same instant, but the two cities are living different parts of the day, and the forecast for "today" is not even quite the same window. A bot that pretended every city was on the same clock would be wrong, sometimes embarrassingly so, every single day.

The third problem was the name thing again, only larger. There are dozens of Springfields. There is a Portland in Oregon and a Portland in Maine. The Germany workaround - editing the cities file by hand and hoping Nominatim picked the right city - was clearly not going to scale to a country where the same name is also a state.

I sat with this for a couple of days before admitting what I already knew.

The bot needed to be rewritten.

What made this hard was not the rewriting itself. It was the requirement to do it without breaking everything else.

By the time I decided to add the United States, the infrastructure around the bot had grown into something I trusted. Jails, snapshots, backup jobs, cron schedules, snac instances on production paths, the HAProxy layer, the homepage cron that aggregated follower counts, and a long list of cities being processed in series every six hours. None of that knew or cared about the bot's internal shape. All of it cared, very much, about the bot's external behavior: a city name and a country code go in, valid markdown comes out, and that markdown ends up in a timeline.

So the contract was clear, even if I had never written it down anywhere. The command-line interface, the output format, the exit codes, the way the wrapper script invoked it, the structure of the JSON country configs - all of it had to keep working. Italian had to keep working. German had to keep working. The cron job that ran every six hours had to keep producing the same shape of output, just with new countries added.

What I changed was almost everything below the surface.

The city argument grew an optional __state suffix, with a double underscore as separator:

python3 main.py springfield__illinois us
python3 main.py springfield__massachusetts us
python3 main.py new_york__new_york us

A city without the suffix continued to work exactly as before, which is what every European country needed. The country config gained a timezone field that could be a fixed string or the literal "auto"; when it was "auto", the bot used timezonefinder against the resolved coordinates to determine the right zone for that specific city. Internally I separated the weather provider behind an interface, so Open-Meteo could remain the primary while MET Norway and wttr.in sat behind as alternatives, with automatic fallback when the primary failed. Units became configurable per country: temperature, wind speed, precipitation. The United States needed Fahrenheit, miles per hour, and inches. Most of Europe wanted Celsius, kilometers per hour, and millimeters. The bot now does either, on a per-country basis, without caring which is which.

I am skipping a lot of small detail here, but the principle was always the same: every new degree of freedom had to be expressible as an optional field in the config or as an optional CLI flag. If a country did not set the new field, the old behavior continued, identical to before.

I tested this by running the new bot against the old country configs and comparing the output line by line. Where it differed, it was a bug in the new bot. Not in the test.

The first cycle after deploying the rewrite was, for every country except the United States, indistinguishable from the cycle before. That was the point.

This is the part of the story I dislike telling, which is precisely why I should tell it.

At some point during the development, while debugging an Open-Meteo response that did not look right, I added a print statement to the error path that dumped the full request URL whenever something went wrong. The full URL of the Open-Meteo customer endpoint includes the apikey query parameter. The print was meant for development. I forgot to remove it.

I deployed.

The next time Open-Meteo had an outage - and small ones happen, sometimes for several minutes at a time - the bot dutifully printed the failing request URL into the post body. For every city. For every cycle that ran during the outage. The wrapper script piped the output into snac note_unlisted without complaint. The posts went out, federated across the Fediverse, with my API key sitting in the text for anyone who cared to read.

Some users were kind enough to write me and tell me. Others were less kind, and made fun of me. Both groups were correct. This should not have happened.

I reported the incident to the Open-Meteo team, who were extremely understanding. They rotated the key immediately and gave me a fresh one. I removed the debug print, and then I did the slightly more useful thing, which was to add redaction at multiple layers - in the bot's output, in the daemon's logging, and in the debug helpers themselves. URL query parameters that look like API keys are masked. Environment variables and config keys named apikey or OPEN_METEO_APIKEY are redacted before any string reaches stdout or a log file. Even JSON-like fields that include open_meteo_apikey are scrubbed if they ever appear in something the program prints.

The lesson is not "be more careful." The lesson is that debug paths leak, sooner or later, so the secrets have to be unreachable from the debug paths in the first place. Now they are.

That afternoon, when I realised what was happening, I closed everything for a minute and looked out of the window. Then I started fixing.

Nominatim is a public service, and it is generous, but it is not infinite. Every city in the project needs coordinates, and at the start of the project every cycle would re-ask Nominatim for every city. Most of the time this worked. Sometimes it did not.

There was one cycle, before I added caching, when Nominatim simply did not respond for one of my queries. The geopy call timed out. The bot raised an exception. The wrapper script gave up on that city and moved on to the next one. A few users noticed that a particular city had not received its forecast that day, and asked what had happened.

I added a coordinate cache, and I am still grateful that I did.

The cache is intentionally boring. The first time the bot resolves a city, it writes the latitude and longitude into a small file under /tmp, named after the city, and the state when present. Every subsequent run reads the file. If the file exists, no Nominatim call is made. If the file is missing, the bot calls Nominatim and writes the file. After the first successful lookup, the cache becomes the source of truth for the coordinates of that city.

This is lighter on Nominatim, faster for every cycle, and much more resilient against transient failures. It is also nice for a reason I did not anticipate.

Nominatim is a geocoder, and like every geocoder it has opinions.

I live in Ferrara, so when I added Italy I made sure Ferrara was in the list, and I checked the first cycle to make sure everything looked right. The forecast came out fine. The temperature was reasonable. The icon matched the sky outside my window. I closed the laptop and forgot about it.

Then, one evening months later, I looked more carefully at the coordinates Nominatim had returned for "Ferrara, Italy", and I realised they did not point to the city. They pointed to a location closer to the centroid of the province, which is a much larger area and mostly countryside. The forecast had been, on average, for a field somewhere outside town, not for the city center.

I am not entirely sure why I had not noticed earlier. Probably because the weather in Ferrara and the weather in the fields outside Ferrara is, on most days, indistinguishable to anyone who is not paying attention. But this is the kind of detail I do not want to leave wrong, especially for my own city.

There are other places where geocoding lands slightly off. Sometimes it is a few kilometers, sometimes a different neighborhood, sometimes genuinely the wrong place.

Because the cache is just a file per city, the fix is also just a file per city. I open the cache file, replace the latitude and longitude with the correct values, save. The next cycle uses the corrected coordinates. No code change, no redeploy, no special tooling. I keep a small list of patched cities in a separate text file, so that if I ever rebuild the cache, I do not lose the manual corrections.

This is the kind of operational simplicity I like. A cache made of plain files costs almost nothing and quietly pays back every time a small problem appears.

For every report it generates, the bot also writes a simplified English text snapshot to /tmp/<city>.txt, or /tmp/<city>__<state>.txt when there is a state.

This is intentional, and it is not a debug artifact. I am not ready to say what I am doing with it yet, but it is part of a future direction for the project. Text is a useful intermediate format, and having a clean, language-neutral representation of every forecast sitting on disk costs almost nothing and might be worth a great deal later.

I prefer to let ideas mature in private before I commit to them in public. So I will leave it at this for the moment.

A full cycle for the United States takes hours.

It is not because the work is heavy. It is because I deliberately inserted a small sleep between cities, to give snac time to dispatch the previous post before the next one is generated. With more than 1200 cities in series, even a short pause adds up. I am not in a hurry. Forecasts that arrive a few minutes apart from each other are not a problem, and the bot was already a polite citizen elsewhere. A polite cycle is fine.

The problem with a slow cycle is not the duration. The problem is what happens to it.

In the original design, the cycle was launched by cron. Every six hours, cron called the wrapper script, the wrapper iterated through the cities file, and for each city it ran the bot and piped the output into snac. There was no scheduler in the project at all. Cron was the scheduler. The wrapper was just a loop.

Restarting snac was harmless. The wrapper would call snac note_unlisted per city, and if snac happened to be unavailable for a moment, that single call might fail, but the loop kept moving and snac was usually back within seconds. Snac itself was not what held the cycle together.

What held the cycle together was the wrapper process. And the wrapper process lived inside the jail.

If the FreeBSD jail was restarted while the wrapper was running, the loop stopped wherever it happened to be. The cron schedule did not care. Six hours later, the next cron tick started a new cycle from the first city, and the cities that had been about to be processed at the moment of the restart were simply skipped for that window. For the United States, this could mean several hundred cities going without an update.

There was a worse case, and it took me longer than it should have to recognise it. If the host was rebooting exactly in the minute when cron should have fired, cron simply did not fire. There was no daemon waiting to pick up the missed tick. The cycle never even started. Six hours of forecasts would be lost, in silence, with nothing in any log to suggest anything had gone wrong.

I lived with this for a long time. Reboots were rare, the impact was limited, and adding state was the kind of thing I always meant to do "next week."

What finally changed it was not a dramatic incident. It was the slow accumulation of small ones. A scheduled VPS reboot. A jail restart after an upgrade. Each one on its own was nothing. Together, they were a steady drip of missed cycles.

So I wrote a daemon.

The crontab entries for the bot went away. There is now a long-running process inside the jail, started at boot, and it does the scheduling itself. The schedule is a list of hours and a minute, read from a JSON config. The daemon wakes up once a minute, checks whether it is time to start a cycle, and either starts one or waits.

The interesting part is the state file.

As the daemon walks through the cities file, it writes its position to a small JSON file: which cities file it is processing, and the index of the next city to handle. The write happens at the boundary between one city and the next, because that is the only place where resuming makes sense. If the daemon is interrupted mid-city, that city is retried on resume; no half-finished post escapes.

When the daemon starts, it reads the state file. If it finds one matching the current cities file, it resumes from the saved index. If the cities file has changed since the state was written, the daemon starts fresh. The check is deliberately conservative: a renamed or modified cities file is treated as a different cycle, because the indices would otherwise be meaningless.

The result is the behavior I should have had from the start. If the host reboots while the United States cycle is running, the daemon comes back up with the jail, reads the state, and continues from where it left off. Every city still gets its update, just with a small gap corresponding to the reboot itself. The cycle finishes. The state file is reset. Life goes on.

And the worst case from the cron days is gone. The daemon does not need anyone to fire it. As long as the jail is running, the daemon is running, and the next scheduled cycle will happen when its hour comes, regardless of what was happening at any specific minute.

Of all the changes I have made to the project, this is the one I like most. It is not exciting work. It is the kind of thing that earns no applause because, when it works, it produces no visible event. But it removes a whole class of small daily annoyances, and it makes a slow process robust against the boring kind of failure: the kind nobody plans for, but that always eventually happens.

The current bot does considerably more than the original Italian script. It handles per-city timezones, three weather providers with automatic fallback, unit conversion for temperature, wind, and precipitation, optional air quality, pressure trend indicators when the provider supplies pressure data, a simplified English text snapshot for future use, a coordinate cache that can be patched by hand, secret redaction at multiple layers, a heartbeat that adapts to whichever HTTP client is installed on the host, and a scheduler-and-resume daemon that survives reboots.

But from the outside, almost nothing has changed.

The European country configs work the same way they always did. The wrapper scripts are unchanged. The snac integration is the same one-line pipe. The HAProxy layer in front does not know or care that the bot was rewritten. The homepage cron that counts followers and regenerates the static page works exactly as before.

The original Italian script does not exist as a file anymore, but it survives as a default. A country config with timezone set to Europe/Rome and no special options behaves, today, exactly as the first version of the bot would have. Everything else is opt-in.

I like this kind of work.

When I wrote about FediMeteo for the first time, I told the story from the beginning: the idea born almost by chance while checking the weather for a holiday, the memory of my grandfather, who for years had been my personal meteorologist, the decision to build something small and useful, and then the surprise of seeing people actually use it. What began as a personal experiment quickly became a small global service, still running with the same philosophy: FreeBSD, jails, simple scripts, snac, text, emoji, and a lot of small pieces doing their work quietly.

That article was mostly about the birth and growth of the project. This one is about one of the less romantic parts of the same story, although I have to admit that I find a certain beauty in it too: keeping the service light as it grows.

FediMeteo is still intentionally simple from the outside. A homepage, some numbers, a list of countries, and many ActivityPub accounts publishing weather forecasts. The posts are text and emoji. There is no JavaScript requirement to read the pages, no heavy frontend, no unnecessary media attached to every forecast, and no dynamic homepage recalculated at every visit just to show the same numbers. This is not accidental. It is the way I wanted the service to behave from the beginning.

But the more the service is used, the more the small details matter. A request that looks harmless when there are ten followers may become a repeated request when there are thousands of followers, remote instances, crawlers, previews, and other servers fetching the same public objects. In the Fediverse, the same small thing can be asked many times by many different places, each one with a perfectly legitimate reason. The backend doesn't care: it just needs to deal with the requests.

And in FediMeteo, the backend is snac.

I like snac very much precisely because it is small, clear, and efficient. It is not a giant application that tries to be everything. It does a focused job and does it well. But this also means that I want to respect its shape. I do not want to waste its threads on work that the reverse proxy can safely do. A snac thread serving the same public avatar again and again is not a tragedy, but it is still a waste. A snac thread answering the same public ActivityPub object several times in the same minute is doing real work, but often not necessary work.

This is the reason behind the HAProxy tuning I am currently using in front of FediMeteo.

It is not about making the configuration look clever. It is about keeping snac quiet.

A continuation of the same idea

I had already explored the same problem with snac and nginx in two previous posts: Improving snac Performance with Nginx Proxy Cache and Caching snac Proxied Media with Nginx. In both cases, the idea was that the reverse proxy should absorb repeated public requests instead of letting them consume snac resources.

This is especially important because snac uses a limited number of threads. I like that. Limits are healthy. They force us to understand what the service is doing, and they prevent a small program from pretending to be an infinite resource. But limits also make waste visible. If a few threads are busy serving files that could have been served from cache, those threads are not available for something more useful.

With FediMeteo the implementation is different because the reverse proxy is HAProxy, but the reasoning is the same. I have many small snac instances, each one in its own FreeBSD (Bastille) jail, and one public entry point that has to route, terminate TLS, compress, cache, and generally remove as much repetitive work as possible from the backends.

This is, in a way, the natural continuation of the original FediMeteo design. In the first article I wrote that I wanted to manage everything according to the Unix philosophy: small pieces working together. This is another piece of that same puzzle. HAProxy does the edge work. snac does the ActivityPub work. Scripts generate forecasts. cron launches updates. ZFS gives me snapshots. FreeBSD jails keep countries separated. Nothing is particularly heroic by itself, but the whole system becomes pleasant because each part has a clear responsibility.

Why there is almost no media

Before talking about HAProxy, it is worth mentioning one of the most important optimizations, which is not in the proxy configuration at all.

FediMeteo does not use media in its forecasts.

No images attached to the posts, no generated weather cards, no maps for each city, no decorative banners. The forecasts are text and emoji. This was a deliberate decision. Weather information does not become more useful just because it is put inside an image, and every media file used by the service would become something to store, serve, cache, federate, expire, back up, and occasionally debug.

Text and emoji are enough. They are accessible, light, readable in text browsers, friendly to timelines, and understandable even when someone does not know the local language perfectly. This was one of the original design principles of FediMeteo, and it also helps the infrastructure. Less media means less work, fewer cache entries, fewer repeated fetches, fewer surprises.

There is one exception: the avatar.

All FediMeteo accounts use the same avatar, and this is also intentional. I could have used a different avatar for each country, or for each city, or created something visually richer. It would have been nicer in some screenshots, perhaps. It would also have been operationally worse.

With one shared avatar, the reverse proxy has one very useful object to cache. It is public, identical for everyone, small, requested often, and therefore almost always hot in cache. HAProxy can serve it directly instead of asking each snac instance to return the same file. Since avatars are requested by remote instances, browsers, profile previews, and all sorts of federation-related fetches, this single decision removes a surprising amount of pointless backend traffic.

So the avatar is not only a visual identity. It is part of the architecture.

This is the kind of optimization I like most, because it starts before the software. It starts with deciding not to create a problem.

The homepage is static because it can be static

The main homepage follows the same logic.

It is a static HTML page generated from a template. Once per hour, a cron script updates the numbers and statistics. It counts the data I want to show, regenerates the page, and then the page remains static until the next run.

This is not because I cannot make a dynamic page. It is because I do not need one. Boring is good.

The homepage does not need to query all the country instances on every visit. It does not need a database request for each user who opens it. It does not need to ask snac anything in real time. The numbers are useful, but they do not need to be updated every second. Once per hour is enough, and it also fits the spirit of the whole project: do the work when it is needed, then serve the result cheaply.

I have seen too many small services become heavy because the first implementation was convenient rather than appropriate. A cron job and a template are not fashionable, but they are often exactly what a page like this needs.

Many countries, one entry point

FediMeteo is made of many country instances. Each one runs in its own jail and listens on its own internal address and port. From the outside, however, they all live under the same domain structure:

fedimeteo.com
www.fedimeteo.com
it.fedimeteo.com
uk.fedimeteo.com
jp.fedimeteo.com
us.fedimeteo.com
usa.fedimeteo.com
can.fedimeteo.com
canada.fedimeteo.com

And many more.

At the beginning, it is always tempting to write one ACL after another in the HAProxy frontend. It is quick, it is explicit, and for five hostnames it is perfectly fine. But FediMeteo did not remain at five hostnames. As countries and aliases grew, a long chain of ACLs would have turned the frontend into a list of names instead of a description of how the proxy behaves.

So I moved the hostname to backend mapping into a map file:

fedimeteo.com        backend_fedimeteo
www.fedimeteo.com    backend_fedimeteo
it.fedimeteo.com     backend_it
uk.fedimeteo.com     backend_uk
jp.fedimeteo.com     backend_jp
us.fedimeteo.com     backend_us
usa.fedimeteo.com    backend_us
can.fedimeteo.com    backend_ca
canada.fedimeteo.com backend_ca

The frontend then needs only one rule:

use_backend %[req.hdr(host),field(1,:),lower,map(/usr/local/etc/fedimeteo.map,backend_fedimeteo)]

This reads the Host header, removes the port if present, lowercases the result, and looks it up in /usr/local/etc/fedimeteo.map. If nothing matches, it falls back to the main FediMeteo backend.

I like this because it keeps the configuration honest. The frontend contains the policy. The map contains the data. Adding a country means adding an entry to the map and defining a backend. I do not need to make the frontend more complicated every time the service grows.

Backends as small compartments

The country backends are deliberately plain:

backend backend_it
    mode http
    http-reuse safe
    server srv1 10.0.0.2:8001 maxconn 30

backend backend_uk
    mode http
    http-reuse safe
    server srv1 10.0.0.7:8001 maxconn 30

backend backend_jp
    mode http
    http-reuse safe
    server srv1 10.0.0.32:8001 maxconn 30

One backend, one jail, one snac instance. This is exactly the same organizational principle as the rest of the project. If I need to reason about Italy, I look at the Italian jail. If I need to reason about the United Kingdom, I look at the UK jail. If one day I need to move a country elsewhere, the separation is already there.

The maxconn 30 value is not a magic number. It is a ceiling. I want each small backend to have a visible limit in front of it. If something starts hammering a country instance, I prefer the pressure to appear at the HAProxy layer instead of becoming unlimited concurrent work inside snac.

http-reuse safe lets HAProxy reuse backend connections where appropriate. This is another small reduction in unnecessary work. Opening connections repeatedly is not the biggest problem in the world, but avoiding it is still better, especially when many small services sit behind the same proxy.

The front door

The HTTPS frontend listens on IPv4 and IPv6 and offers both HTTP/2 and HTTP/1.1:

frontend https_in
    bind :::443 v4v6 ssl crt /usr/local/etc/certs/ alpn h2,http/1.1
    mode http
    option http-keep-alive

TLS defaults are set globally:

ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

Port 80 only redirects to HTTPS, except for Let's Encrypt challenges:

acl letsencrypt-acl path_beg /.well-known/acme-challenge/
http-request redirect scheme https code 301 unless letsencrypt-acl
use_backend letsencrypt-backend if letsencrypt-acl

In the HTTPS frontend I also set the usual forwarding headers:

http-request set-header X-Real-IP %[src]
http-request set-header X-Forwarded-Proto https

And I add HSTS:

http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"

None of this is unusual, and that is fine. The interesting parts of an infrastructure are not always the parts that should be unusual.

Two caches, because the requests are different

The HAProxy configuration defines two caches:

cache mediacache
  total-max-size 128
  max-object-size 10000000
  max-age 3600
  process-vary on
  max-secondary-entries 12

cache jsoncache
  total-max-size 16
  max-object-size 1000000
  max-age 60
  process-vary on
  max-secondary-entries 12

I keep media and ActivityPub JSON separate because they are not the same kind of traffic.

The media cache is larger and has a longer maximum age. In FediMeteo, this mostly means the shared avatar and a few static-looking objects. Since there is intentionally almost no media, the important cached object is requested very often and remains warm.

The JSON cache is smaller and short-lived. It is there for public ActivityPub GET requests, not to store federation state forever. A 60 second cache is enough to collapse many repeated requests that arrive close together in time, without pretending that ActivityPub responses should be treated like immutable files.

This distinction is important. Caching is not one decision. It is a set of small decisions about what a response means, who can see it, how often it changes, and what happens if it is served again.

Recognizing media

For media, the ACL is based on file extensions:

acl is_media path_end -i .jpg .jpeg .png .gif .webp .svg .ico .mp4 .webm .mp3 .ogg .wav .flac .mov .avi .mkv .m4v

Then I store the result in a transaction variable:

http-request set-var(txn.is_media) bool(true) if is_media

The cache lookup is straightforward:

http-request cache-use mediacache if { var(txn.is_media) -m bool true }

And on the response side:

http-response set-header Cache-Control "max-age=3600, public" if { var(txn.is_media) -m bool true }
http-response del-header Set-Cookie if { var(txn.is_media) -m bool true }
http-response del-header Vary if { var(txn.is_media) -m bool true }
http-response cache-store mediacache if { var(txn.is_media) -m bool true }

The Cache-Control header makes the intent explicit. Set-Cookie is removed because a public media object should not carry session information. Vary is removed because I do not want the same avatar to fragment into many cache entries because of harmless header differences.

This is aggressive only if removed from its context. In this service, with this media policy, it is a reasonable choice. FediMeteo is not serving private media under these paths. It is mostly serving the same public avatar over and over.

For the same reason, I clean the request before it reaches the backend:

http-request del-header Authorization if { var(txn.is_media) -m bool true }
http-request del-header Cookie        if { var(txn.is_media) -m bool true }

I would not do this globally. I do it after deciding that the request is media. Scope is what makes these rules safe.

The result is exactly what I want: the shared avatar becomes an almost perfect cache object. Small, public, repeatedly requested, and served by HAProxy instead of snac.

ActivityPub JSON microcaching

The ActivityPub side starts from the Accept header:

acl is_ap_json   req.hdr(Accept),lower -m sub application/activity+json
acl is_ap_ldjson req.hdr(Accept),lower -m sub application/ld+json
acl is_outbox    path_end /outbox
acl is_get       method GET
acl has_auth     req.hdr(Authorization) -m found
acl has_cookie   req.hdr(Cookie) -m found

This part matters because ActivityPub uses content negotiation. The same path may return HTML to a browser and JSON to a remote instance. If the proxy pretends that a URL is always one thing, it will eventually cache the wrong representation.

So I only mark public ActivityPub GET requests as cacheable:

http-request set-var(txn.is_activitypub) bool(true) if is_get !is_outbox is_ap_json !has_auth !has_cookie
http-request set-var(txn.is_activitypub) bool(true) if is_get !is_outbox is_ap_ldjson !has_auth !has_cookie

There are several decisions here, all important.

It must be a GET, because I am not caching deliveries or anything that changes state. It must not be /outbox, because outbox collections are not the traffic I want to cache here. It must not have Authorization, and it must not have cookies, because authenticated or user-specific requests do not belong in a shared public cache.

Then the cache can be used and populated:

http-request cache-use jsoncache if { var(txn.is_activitypub) -m bool true }

http-response set-header Cache-Control "max-age=60, public" if { var(txn.is_activitypub) -m bool true }
http-response cache-store jsoncache if { var(txn.is_activitypub) -m bool true }

Sixty seconds is short, but useful. Federation often creates small clusters of identical requests. A remote server fetches an actor, another fetches the same actor, something asks for the same object, something retries. I do not need to cache these responses for hours. I only need HAProxy to answer the second and third identical request during the same small burst.

This is microcaching in the most practical sense. It reduces repeated work without changing the nature of the service.

Static media paths

There is also a rule for static paths:

acl is_short_path path_reg ^/[^/]+/s/
http-request cache-use mediacache if is_short_path

This comes from the same observation that led me to cache snac media with nginx. snac uses static media paths, and those paths often represent the kind of public, repeatable traffic that should not consume backend threads if the proxy can serve it. I call them "short", not because they are, but because the first time I saw them, I thought the 's' stood for "short", not "static". The name just stuck.

In FediMeteo this is less central than on a normal social instance, because I deliberately do not use media except for the avatar and basic static objects. Still, the rule fits the general policy: let HAProxy handle repeatable edge work, and let snac spend its threads where they are actually needed.

Vary, but not without limits

Both caches have:

process-vary on
max-secondary-entries 12

I want HAProxy to process Vary, because content negotiation is real, especially when ActivityPub is involved. But I also want variation to be bounded. If every slightly different header creates another cache entry, the cache becomes a complicated way to miss.

For media, I remove Vary before storing the response. A shared avatar does not need to vary by Accept. For ActivityPub JSON, I am more careful because the representation matters.

Again, the important thing is not the number itself. It is the decision to make variation explicit and limited.

Seeing whether it works

During rollout, I like to expose a very small diagnostic header:

http-response set-header X-Cache-Status HIT if !{ srv_id -m found }
http-response set-header X-Cache-Status MISS if { srv_id -m found }

This is intentionally simple. If HAProxy selected a backend server, I call it a miss. If no backend server was selected, the response came from cache, so I call it a hit. It is not a complete observability system, but it is enough to answer the first question I usually have after changing a cache rule.

Did this request reach snac?

A test can be as simple as:

curl -I https://it.fedimeteo.com/path/to/avatar.png
curl -I https://it.fedimeteo.com/path/to/avatar.png

The second request should be a hit.

For ActivityPub JSON, the test must use the right Accept header:

curl -I \
  -H 'Accept: application/activity+json' \
  https://it.fedimeteo.com/some/activitypub/object

And I also want to verify that cookies and authorization prevent public caching:

curl -I \
  -H 'Cookie: test=value' \
  -H 'Accept: application/activity+json' \
  https://it.fedimeteo.com/some/activitypub/object

curl -I \
  -H 'Authorization: Bearer fake' \
  -H 'Accept: application/activity+json' \
  https://it.fedimeteo.com/some/activitypub/object

A cache that works should be visible. A cache that is invisible can be correct, but it can also be silently wrong. I prefer to know.

Compression and operational paths

HAProxy also handles gzip compression:

filter compression
compression algo gzip
compression type text/css text/html text/javascript application/javascript text/plain text/xml application/json application/activity+json

This keeps another common responsibility at the edge. The country instances can stay focused on snac and the forecast data, while HAProxy deals with client-facing compression for HTML, JSON, and ActivityPub responses.

There is also a local Prometheus exporter:

frontend prometheus
  bind 127.0.0.1:8405
  mode http
  http-request use-service prometheus-exporter
  no log

And I keep internal operational paths, such as statistics and Grafana, handled before the hostname map. These are small details, but ordering matters. Special paths should be explicit and early. The hostname map is for FediMeteo routing, not for every internal tool I happen to expose behind the same proxy.

What this changes in practice

The nice thing about this configuration is that none of its parts is particularly surprising.

The map keeps hostname routing manageable. The backend definitions keep each country isolated and limited. The static homepage avoids dynamic work for something that changes once per hour. The shared avatar gives HAProxy one very hot media object to serve directly. The media cache keeps public files away from snac. The JSON microcache absorbs short ActivityPub bursts. Header cleanup prevents useless variation. Connection reuse avoids unnecessary backend connection churn.

But all of this is only a longer way of saying one thing:

fewer requests reach snac.

That is the metric I care about here.

Not because snac is slow. If anything, FediMeteo exists in its current form because snac is efficient enough to make this kind of project possible on a very small VPS. But precisely because the whole architecture is small and pleasant, I do not want to waste resources where there is no need.

This is also consistent with the rest of the project. Forecasts are serialized by scripts. Updates happen every six hours. The homepage is regenerated hourly. Countries live in separate jails. Snapshots and backups are handled outside the application. No single component tries to be the entire system.

HAProxy is just another small piece, but it sits in the right place to remove a lot of repeated work.

Caveats

This configuration is not a universal HAProxy recipe for ActivityPub services.

It matches FediMeteo as it is now: almost no media, one shared avatar, static homepage, public forecasts, many small snac instances, and ActivityPub traffic that can benefit from a short public cache when there are no cookies or authorization headers.

If I decide one day to use media in forecasts, the media cache rules will need to be reviewed. If I use different avatars for each city or country, the cache will still work, but I will lose the very nice property of one shared, always-hot avatar. If ActivityPub responses become actor-dependent, public JSON caching must be reconsidered. If one country grows a very different traffic pattern from the others, it may deserve a different limit or policy.

This is why I do not like presenting configurations as magic. A good configuration is a written form of the assumptions behind a service. When the assumptions change, the configuration must change too.

Conclusion

FediMeteo started as a small idea and became larger than I expected, but I still want it to feel small in the right ways. Small does not mean fragile. Small means understandable. It means that each part has a reason to exist, and that unnecessary work is removed before it becomes a problem.

The HAProxy layer follows this idea. It terminates TLS, routes hostnames through a map, reuses backend connections, serves the shared avatar from cache, microcaches public ActivityPub JSON, avoids authenticated and cookie-based traffic, and gives me a small diagnostic header to see what is happening.

There is no single brilliant directive here. There is only the usual work of matching infrastructure to reality.

FediMeteo publishes weather forecasts as text and emoji. The homepage is static HTML updated every hour. The accounts share the same avatar because it is enough, and because it is better for the cache. Each country has its own snac instance in its own FreeBSD jail. HAProxy stands in front of them and tries, quietly, not to bother them unless it has to.

I like this kind of infrastructure.

Not because it is invisible, but because when it works well, it leaves very little to say.

LibreNMS has been a faithful companion for years now. It quietly handles the monitoring of my servers, devices, and services without demanding much in return - exactly what you want from a tool whose job is to watch over everything else. It's a solid alternative to heavier solutions like Zabbix, and it gives you alerts, data, and graphs on virtually anything reachable over SNMP.

I usually install it on a host that is not reachable from the outside, then let it poll all the devices through a VPN: a single observation point, clean perimeter. The ability to create multiple dashboards - and to filter them by user - has also let me give clients a transparent window onto their own servers. Transparency, in my experience, is always the better long-term bet.

Together with Uptime-Kuma (and the good old Nagios/Munin pair), LibreNMS lives in a FreeBSD jail on my monitoring servers and just does its job.

This post walks through a plain installation of LibreNMS on FreeBSD: package-based, no reverse proxy, no HTTPS, no fancy hardening. The goal is to get to a working setup you can build on top of.

Assumptions

• FreeBSD 15.0-RELEASE, in a jail or on a dedicated VM/host
• nginx + php-fpm + MySQL 8.4
• LibreNMS installed from the official package — not via git clone

One note before we start: in this guide I use plain HTTP just to reach the first-time setup. If your LibreNMS instance won't stay confined to a private network or behind a VPN, configuring HTTPS is mandatory, not optional.

Installation

pkg install librenms mysql84-server python3 nginx

LibreNMS currently depends on PHP 8.4. If you want to speed PHP up, install OPcache too:

pkg install php84-opcache

MySQL

Two settings need to be in place before MySQL starts for the first time. After the first start they cannot be changed without reinitializing the data directory, so it's worth getting them right now.

cd /usr/local/etc/mysql
cp my.cnf.sample my.cnf

In the [mysqld] section, add:

innodb_file_per_table=1
lower_case_table_names=0

Now start MySQL:

service mysql-server enable
service mysql-server start

On a fresh FreeBSD install, the local root user can connect to MySQL without a password from the command line. Connect and create the database and user. I'm using password here as a placeholder - don't.

mysql

CREATE DATABASE librenms CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'librenms'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON librenms.* TO 'librenms'@'localhost';
exit

php-fpm

Edit /usr/local/etc/php-fpm.d/www.conf and adjust the listen directives:

listen = /var/run/php-fpm-librenms.sock
listen.owner = www
listen.group = www
listen.mode = 0660

Then create php.ini from the production sample:

cd /usr/local/etc
cp php.ini-production php.ini

And set the timezone in php.ini:

date.timezone = Europe/Rome

nginx

Since this jail (or host) is dedicated to LibreNMS, we can rewrite the server block in /usr/local/etc/nginx/nginx.conf directly:

server {
    listen      80;
    #server_name yourServerName
    root        /usr/local/www/librenms/html;
    index       index.php;

    charset utf-8;
    gzip on;
    gzip_types text/css application/javascript text/javascript application/x-javascript image/svg+xml text/plain text/xsd text/xsl text/xml image/x-icon;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location /api/v0 {
        try_files $uri $uri/ /api_v0.php?$query_string;
    }

    location ~ \.php$ {
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        set $path_info $fastcgi_path_info;
        try_files $fastcgi_script_name =404;
        include fastcgi_params;
        fastcgi_param SERVER_SOFTWARE "";
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $path_info;
        fastcgi_index index.php;
        fastcgi_pass unix:/var/run/php-fpm-librenms.sock;
        fastcgi_buffers 256 4k;
        fastcgi_intercept_errors on;
        fastcgi_read_timeout 14400;
    }

    location ~ /\.(?!well-known).* {
        deny all;
    }
}

Now start nginx and php-fpm:

service nginx enable
service nginx start

service php_fpm enable
service php_fpm start

LibreNMS configuration

Copy the default config:

cp /usr/local/www/librenms/config.php.default /usr/local/www/librenms/config.php

Because we installed from the package, this file already has the right commands and paths for FreeBSD - no need to hunt down mtr, fping, snmpwalk and friends one by one.

Create the directory for RRD graphs and set ownership:

mkdir -p /var/db/librenms/rrd
chown -R www:www /var/db/librenms
chmod 775 /var/db/librenms/rrd

Then the .env file:

cd /usr/local/www/librenms
cp .env.example .env
chown www .env

Edit .env and set at least:

• DB_DATABASE - librenms
• DB_USERNAME - librenms
• DB_PASSWORD - the one you actually used (not password, please)

Then add this line, which tells LibreNMS we still need to run the web installer:

INSTALL=true

A note on permissions. The official LibreNMS documentation suggests chown -R www:www over the entire application tree, but on FreeBSD the package already lays down sane ownership, with storage/ and bootstrap/cache/ writable by www. There's no reason to widen the rest of the codebase. If validate.php complains later about something write-related, the first place to check is:

ls -la /usr/local/www/librenms/storage /usr/local/www/librenms/bootstrap/cache

Now generate the app key as www, since the file is owned by www:

su -m www -c "php artisan key:generate"

And tighten .env:

chmod 600 .env

Refresh the configuration cache:

su -m www -c "lnms config:clear"
su -m www -c "lnms config:cache"

Web installer

Open http://host/install and follow the steps. The validation process may fail. Refreshing the cache picks up the values written to config.php during the install:

su -m www -c "lnms config:clear"
su -m www -c "lnms config:cache"

When the web installer is done, edit .env again and remove the INSTALL=true line if it's still there. Leaving it in place re-exposes the installer to anyone who can reach the URL.

Polling service

LibreNMS needs something to actually run the polls. On FreeBSD, the package ships an rc service that runs the LibreNMS dispatcher, so there's no need to manage cron entries by hand the way most Linux guides assume.

service librenms enable
service librenms start

Validate

cd /usr/local/www/librenms
su -m www -c './validate.php'

You may see a couple of complaints right after starting the service - usually scheduler-related and self-resolving within a few minutes. Re-run validate.php once the dispatcher has had time to settle. Anything still red after that is worth investigating.

Next steps

At this point you can log into the web interface and start adding devices, configuring SNMP, and building dashboards. For that, the official LibreNMS documentation is excellent, and there's no point in me paraphrasing it here.

Mastodon 4.5.0 is here, and with it come some interesting changes that, in my opinion, might encourage more people to consider it for self-hosting their Fediverse community.

While it may not be as lightweight and simple as other solutions (like snac or GoToSocial or many others), I believe it remains one of the best platforms for managing a medium-sized Fediverse community, thanks in part to the direct feedback that many admins have provided to the developers.

I have previously written about how to install Mastodon in a FreeBSD jail and how to modify its character and poll limits.

One of the most critical initial decisions (which can be changed later, but with extra work) is where to store your media files. Mastodon downloads and re-processes all media it encounters from other instances for three main reasons:

• Local Caching: Your users connect to your media server, reducing the load on the original instance.
• Security: Re-processing media helps to remove any potential "impurities" before they reach the user's device.
• Privacy: It prevents disclosing your users' IP addresses to other instances. A user will only connect to their own instance to fetch all data, including remote content.

At least initially, media files will be the largest part of your instance's storage footprint. It is therefore essential to plan where to store them and to add a regular cleanup script; otherwise, their growth will be exponential.

Mastodon supports uploading media to external S3-compatible solutions, and many admins use the usual commercial providers, paying for data uploads and transfers.

I am a firm believer in "Own Your Data", so I have always used my own self-hosted S3 servers. I initially started with Minio, but over time, I realized that, by design, it doesn't perform well with a multitude of small files (performance degrades). After running some tests, I decided to switch to SeaweedFS.

SeaweedFS "is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek..." - this, combined with the fact that it is a mature and proven piece of software, was enough for me to give it a try. The result? Excellent. The I/O and CPU load on my media server dropped drastically, making SeaweedFS an incredibly suitable solution. Furthermore, some of its features (like the ability to run a filer.sync) allow for efficient and fast replication to other storage, another host, or... anything else.

SeaweedFS works perfectly with Mastodon, and I will explain the steps to get it into production.

I will install SeaweedFS in a dedicated jail and use a dedicated subdomain. This ensures that the media server can be moved to another host at any time without reconfiguring everything or changing domains. SeaweedFS has its own FreeBSD package, installable via pkg, or can be downloaded directly from the project's website.

In either case, I will describe a "test" setup - which can also be used in production without issues. However, I highly recommend diving deeper into the tool, as it is incredibly powerful and flexible and can solve many more problems than one might imagine.

Setting up the SeaweedFS Jail

First, let's create a dedicated jail with BastilleBSD:

bastille create media 14.3-RELEASE 10.0.0.66 bastille0

Now, let's enter the jail and install SeaweedFS (and tmux, which can be useful):

bastille console media
pkg install -y tmux seaweedfs

I suggest launching SeaweedFS in a tmux session so you can monitor its output. Later, you should configure an automatic startup method, such as using the included rc.d file or any other method you prefer.

Create a directory for the data and start SeaweedFS as the "seaweedfs" user:

mkdir -p /seaweedfs/data
chown -R seaweedfs /seaweedfs
su -m seaweedfs
cd /seaweedfs/
/usr/local/bin/weed server -dir /seaweedfs/data -s3

At this point, SeaweedFS will start and create everything it needs to function, including the S3 server.

Configuring Buckets and Users

Now, let's open the weed shell to create the necessary bucket and users:

weed shell
s3.bucket.create -name mastomedia

Still in the weed shell, create a user for Mastodon and grant read permissions for unauthenticated users (which is necessary to serve media to the world):

s3.configure -access_key=mastomedia -secret_key=CHANGEME -buckets=mastomedia -user=mastodon -actions=Read,Write,List,Tagging,Admin -apply
s3.configure -buckets=mastomedia -user=anonymous -actions=Read -apply
s3.configure -buckets=mastomedia -actions=Read -apply

Security Tip: For the -secret_key, avoid using a simple password. You can generate a strong, random key directly from your shell with a command like openssl rand -base64 32.

Done. SeaweedFS is now ready to receive (and serve) media. The next step is to set up a reverse proxy to serve everything over HTTPS. My preferred approach is to configure the system as if it were external, even if the services are in adjacent jails. This might use slightly more resources, but the time and trouble it saves in the future are well worth it.

Nginx Reverse Proxy Configuration

The reverse proxy can be configured something like this:

[...]

server {
   server_name  media.mastodon.example.com;

   ignore_invalid_headers off;
   client_max_body_size 0; # Allow large file uploads without Nginx limits

   location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_connect_timeout 300;
      proxy_http_version 1.1;
      proxy_set_header Connection "";
      chunked_transfer_encoding off;

      expires 1y;
      add_header Cache-Control public;

      add_header X-Cache-Status $upstream_cache_status;
      add_header X-Content-Type-Options nosniff;

      proxy_pass http://10.0.0.66:8333;
   }

# ... other server configurations like SSL ...

}

Mastodon Configuration

Now let's configure Mastodon. If you are running the setup wizard for the first time, here is a summary of the options:

[...]
Do you want to store uploaded files on the cloud? yes
Provider Minio
Minio endpoint URL: https://media.mastodon.example.com
Minio bucket name: mastomedia
Minio access key: mastomedia
Minio secret key: CHANGEME
Do you want to access the uploaded files from your own domain? Yes
Domain for uploaded files: media.mastodon.example.com

If Mastodon is already active, or once the setup is complete, the options in your .env.prod file should be modified to be consistent with what SeaweedFS expects:

S3_ENABLED=true
S3_PROTOCOL=https
S3_REGION=us-east-1
S3_ENDPOINT=https://media.mastodon.example.com
S3_HOSTNAME=media.mastodon.example.com
S3_BUCKET=mastomedia
AWS_ACCESS_KEY_ID=mastomedia
AWS_SECRET_ACCESS_KEY=CHANGEME
S3_FORCE_SINGLE_REQUEST=true
# remove the S3_ALIAS_HOST if it is set

IMPORTANT NOTE: If both services are in jails on the same host (i.e., SeaweedFS is on the same host as Mastodon), you should ensure that the Mastodon jail can reach the SeaweedFS jail through the reverse proxy and not via the external IP. To do this, add the following line to the /etc/hosts file of the Mastodon jail:

10.0.0.1        media.mastodon.example.com

In this example, the reverse proxy is at 10.0.0.1. If you are not using a separate reverse proxy but are exposing Nginx directly from the jail (as described in my Mastodon installation article), use the IP of the Mastodon jail itself instead (e.g., 10.0.0.42).

With this setup, Mastodon will be able to upload media to the SeaweedFS server and generate the correct links for other instances, public visitors, and users of your own instance.

Have fun with SeaweedFS!

How a simple idea turned into an international weather service on the Fediverse.

Using nginx to cache proxied media in snac to improve performance and privacy

Using nginx to cache snac multimedia files to boost performance