When I wrote about FediMeteo for the first time, I told the story from the beginning: the idea born almost by chance while checking the weather for a holiday, the memory of my grandfather, who for years had been my personal meteorologist, the decision to build something small and useful, and then the surprise of seeing people actually use it. What began as a personal experiment quickly became a small global service, still running with the same philosophy: FreeBSD, jails, simple scripts, snac, text, emoji, and a lot of small pieces doing their work quietly.

That article was mostly about the birth and growth of the project. This one is about one of the less romantic parts of the same story, although I have to admit that I find a certain beauty in it too: keeping the service light as it grows.

FediMeteo is still intentionally simple from the outside. A homepage, some numbers, a list of countries, and many ActivityPub accounts publishing weather forecasts. The posts are text and emoji. There is no JavaScript requirement to read the pages, no heavy frontend, no unnecessary media attached to every forecast, and no dynamic homepage recalculated at every visit just to show the same numbers. This is not accidental. It is the way I wanted the service to behave from the beginning.

But the more the service is used, the more the small details matter. A request that looks harmless when there are ten followers may become a repeated request when there are thousands of followers, remote instances, crawlers, previews, and other servers fetching the same public objects. In the Fediverse, the same small thing can be asked many times by many different places, each one with a perfectly legitimate reason. The backend doesn't care: it just needs to deal with the requests.

And in FediMeteo, the backend is snac.

I like snac very much precisely because it is small, clear, and efficient. It is not a giant application that tries to be everything. It does a focused job and does it well. But this also means that I want to respect its shape. I do not want to waste its threads on work that the reverse proxy can safely do. A snac thread serving the same public avatar again and again is not a tragedy, but it is still a waste. A snac thread answering the same public ActivityPub object several times in the same minute is doing real work, but often not necessary work.

This is the reason behind the HAProxy tuning I am currently using in front of FediMeteo.

It is not about making the configuration look clever. It is about keeping snac quiet.

A continuation of the same idea

I had already explored the same problem with snac and nginx in two previous posts: Improving snac Performance with Nginx Proxy Cache and Caching snac Proxied Media with Nginx. In both cases, the idea was that the reverse proxy should absorb repeated public requests instead of letting them consume snac resources.

This is especially important because snac uses a limited number of threads. I like that. Limits are healthy. They force us to understand what the service is doing, and they prevent a small program from pretending to be an infinite resource. But limits also make waste visible. If a few threads are busy serving files that could have been served from cache, those threads are not available for something more useful.

With FediMeteo the implementation is different because the reverse proxy is HAProxy, but the reasoning is the same. I have many small snac instances, each one in its own FreeBSD (Bastille) jail, and one public entry point that has to route, terminate TLS, compress, cache, and generally remove as much repetitive work as possible from the backends.

This is, in a way, the natural continuation of the original FediMeteo design. In the first article I wrote that I wanted to manage everything according to the Unix philosophy: small pieces working together. This is another piece of that same puzzle. HAProxy does the edge work. snac does the ActivityPub work. Scripts generate forecasts. cron launches updates. ZFS gives me snapshots. FreeBSD jails keep countries separated. Nothing is particularly heroic by itself, but the whole system becomes pleasant because each part has a clear responsibility.

Why there is almost no media

Before talking about HAProxy, it is worth mentioning one of the most important optimizations, which is not in the proxy configuration at all.

FediMeteo does not use media in its forecasts.

No images attached to the posts, no generated weather cards, no maps for each city, no decorative banners. The forecasts are text and emoji. This was a deliberate decision. Weather information does not become more useful just because it is put inside an image, and every media file used by the service would become something to store, serve, cache, federate, expire, back up, and occasionally debug.

Text and emoji are enough. They are accessible, light, readable in text browsers, friendly to timelines, and understandable even when someone does not know the local language perfectly. This was one of the original design principles of FediMeteo, and it also helps the infrastructure. Less media means less work, fewer cache entries, fewer repeated fetches, fewer surprises.

There is one exception: the avatar.

All FediMeteo accounts use the same avatar, and this is also intentional. I could have used a different avatar for each country, or for each city, or created something visually richer. It would have been nicer in some screenshots, perhaps. It would also have been operationally worse.

With one shared avatar, the reverse proxy has one very useful object to cache. It is public, identical for everyone, small, requested often, and therefore almost always hot in cache. HAProxy can serve it directly instead of asking each snac instance to return the same file. Since avatars are requested by remote instances, browsers, profile previews, and all sorts of federation-related fetches, this single decision removes a surprising amount of pointless backend traffic.

So the avatar is not only a visual identity. It is part of the architecture.

This is the kind of optimization I like most, because it starts before the software. It starts with deciding not to create a problem.

The homepage is static because it can be static

The main homepage follows the same logic.

It is a static HTML page generated from a template. Once per hour, a cron script updates the numbers and statistics. It counts the data I want to show, regenerates the page, and then the page remains static until the next run.

This is not because I cannot make a dynamic page. It is because I do not need one. Boring is good.

The homepage does not need to query all the country instances on every visit. It does not need a database request for each user who opens it. It does not need to ask snac anything in real time. The numbers are useful, but they do not need to be updated every second. Once per hour is enough, and it also fits the spirit of the whole project: do the work when it is needed, then serve the result cheaply.

I have seen too many small services become heavy because the first implementation was convenient rather than appropriate. A cron job and a template are not fashionable, but they are often exactly what a page like this needs.

Many countries, one entry point

FediMeteo is made of many country instances. Each one runs in its own jail and listens on its own internal address and port. From the outside, however, they all live under the same domain structure:

fedimeteo.com
www.fedimeteo.com
it.fedimeteo.com
uk.fedimeteo.com
jp.fedimeteo.com
us.fedimeteo.com
usa.fedimeteo.com
can.fedimeteo.com
canada.fedimeteo.com

And many more.

At the beginning, it is always tempting to write one ACL after another in the HAProxy frontend. It is quick, it is explicit, and for five hostnames it is perfectly fine. But FediMeteo did not remain at five hostnames. As countries and aliases grew, a long chain of ACLs would have turned the frontend into a list of names instead of a description of how the proxy behaves.

So I moved the hostname to backend mapping into a map file:

fedimeteo.com        backend_fedimeteo
www.fedimeteo.com    backend_fedimeteo
it.fedimeteo.com     backend_it
uk.fedimeteo.com     backend_uk
jp.fedimeteo.com     backend_jp
us.fedimeteo.com     backend_us
usa.fedimeteo.com    backend_us
can.fedimeteo.com    backend_ca
canada.fedimeteo.com backend_ca

The frontend then needs only one rule:

use_backend %[req.hdr(host),field(1,:),lower,map(/usr/local/etc/fedimeteo.map,backend_fedimeteo)]

This reads the Host header, removes the port if present, lowercases the result, and looks it up in /usr/local/etc/fedimeteo.map. If nothing matches, it falls back to the main FediMeteo backend.

I like this because it keeps the configuration honest. The frontend contains the policy. The map contains the data. Adding a country means adding an entry to the map and defining a backend. I do not need to make the frontend more complicated every time the service grows.

Backends as small compartments

The country backends are deliberately plain:

backend backend_it
    mode http
    http-reuse safe
    server srv1 10.0.0.2:8001 maxconn 30

backend backend_uk
    mode http
    http-reuse safe
    server srv1 10.0.0.7:8001 maxconn 30

backend backend_jp
    mode http
    http-reuse safe
    server srv1 10.0.0.32:8001 maxconn 30

One backend, one jail, one snac instance. This is exactly the same organizational principle as the rest of the project. If I need to reason about Italy, I look at the Italian jail. If I need to reason about the United Kingdom, I look at the UK jail. If one day I need to move a country elsewhere, the separation is already there.

The maxconn 30 value is not a magic number. It is a ceiling. I want each small backend to have a visible limit in front of it. If something starts hammering a country instance, I prefer the pressure to appear at the HAProxy layer instead of becoming unlimited concurrent work inside snac.

http-reuse safe lets HAProxy reuse backend connections where appropriate. This is another small reduction in unnecessary work. Opening connections repeatedly is not the biggest problem in the world, but avoiding it is still better, especially when many small services sit behind the same proxy.

The front door

The HTTPS frontend listens on IPv4 and IPv6 and offers both HTTP/2 and HTTP/1.1:

frontend https_in
    bind :::443 v4v6 ssl crt /usr/local/etc/certs/ alpn h2,http/1.1
    mode http
    option http-keep-alive

TLS defaults are set globally:

ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

Port 80 only redirects to HTTPS, except for Let's Encrypt challenges:

acl letsencrypt-acl path_beg /.well-known/acme-challenge/
http-request redirect scheme https code 301 unless letsencrypt-acl
use_backend letsencrypt-backend if letsencrypt-acl

In the HTTPS frontend I also set the usual forwarding headers:

http-request set-header X-Real-IP %[src]
http-request set-header X-Forwarded-Proto https

And I add HSTS:

http-response set-header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"

None of this is unusual, and that is fine. The interesting parts of an infrastructure are not always the parts that should be unusual.

Two caches, because the requests are different

The HAProxy configuration defines two caches:

cache mediacache
  total-max-size 128
  max-object-size 10000000
  max-age 3600
  process-vary on
  max-secondary-entries 12

cache jsoncache
  total-max-size 16
  max-object-size 1000000
  max-age 60
  process-vary on
  max-secondary-entries 12

I keep media and ActivityPub JSON separate because they are not the same kind of traffic.

The media cache is larger and has a longer maximum age. In FediMeteo, this mostly means the shared avatar and a few static-looking objects. Since there is intentionally almost no media, the important cached object is requested very often and remains warm.

The JSON cache is smaller and short-lived. It is there for public ActivityPub GET requests, not to store federation state forever. A 60 second cache is enough to collapse many repeated requests that arrive close together in time, without pretending that ActivityPub responses should be treated like immutable files.

This distinction is important. Caching is not one decision. It is a set of small decisions about what a response means, who can see it, how often it changes, and what happens if it is served again.

Recognizing media

For media, the ACL is based on file extensions:

acl is_media path_end -i .jpg .jpeg .png .gif .webp .svg .ico .mp4 .webm .mp3 .ogg .wav .flac .mov .avi .mkv .m4v

Then I store the result in a transaction variable:

http-request set-var(txn.is_media) bool(true) if is_media

The cache lookup is straightforward:

http-request cache-use mediacache if { var(txn.is_media) -m bool true }

And on the response side:

http-response set-header Cache-Control "max-age=3600, public" if { var(txn.is_media) -m bool true }
http-response del-header Set-Cookie if { var(txn.is_media) -m bool true }
http-response del-header Vary if { var(txn.is_media) -m bool true }
http-response cache-store mediacache if { var(txn.is_media) -m bool true }

The Cache-Control header makes the intent explicit. Set-Cookie is removed because a public media object should not carry session information. Vary is removed because I do not want the same avatar to fragment into many cache entries because of harmless header differences.

This is aggressive only if removed from its context. In this service, with this media policy, it is a reasonable choice. FediMeteo is not serving private media under these paths. It is mostly serving the same public avatar over and over.

For the same reason, I clean the request before it reaches the backend:

http-request del-header Authorization if { var(txn.is_media) -m bool true }
http-request del-header Cookie        if { var(txn.is_media) -m bool true }

I would not do this globally. I do it after deciding that the request is media. Scope is what makes these rules safe.

The result is exactly what I want: the shared avatar becomes an almost perfect cache object. Small, public, repeatedly requested, and served by HAProxy instead of snac.

ActivityPub JSON microcaching

The ActivityPub side starts from the Accept header:

acl is_ap_json   req.hdr(Accept),lower -m sub application/activity+json
acl is_ap_ldjson req.hdr(Accept),lower -m sub application/ld+json
acl is_outbox    path_end /outbox
acl is_get       method GET
acl has_auth     req.hdr(Authorization) -m found
acl has_cookie   req.hdr(Cookie) -m found

This part matters because ActivityPub uses content negotiation. The same path may return HTML to a browser and JSON to a remote instance. If the proxy pretends that a URL is always one thing, it will eventually cache the wrong representation.

So I only mark public ActivityPub GET requests as cacheable:

http-request set-var(txn.is_activitypub) bool(true) if is_get !is_outbox is_ap_json !has_auth !has_cookie
http-request set-var(txn.is_activitypub) bool(true) if is_get !is_outbox is_ap_ldjson !has_auth !has_cookie

There are several decisions here, all important.

It must be a GET, because I am not caching deliveries or anything that changes state. It must not be /outbox, because outbox collections are not the traffic I want to cache here. It must not have Authorization, and it must not have cookies, because authenticated or user-specific requests do not belong in a shared public cache.

Then the cache can be used and populated:

http-request cache-use jsoncache if { var(txn.is_activitypub) -m bool true }

http-response set-header Cache-Control "max-age=60, public" if { var(txn.is_activitypub) -m bool true }
http-response cache-store jsoncache if { var(txn.is_activitypub) -m bool true }

Sixty seconds is short, but useful. Federation often creates small clusters of identical requests. A remote server fetches an actor, another fetches the same actor, something asks for the same object, something retries. I do not need to cache these responses for hours. I only need HAProxy to answer the second and third identical request during the same small burst.

This is microcaching in the most practical sense. It reduces repeated work without changing the nature of the service.

Static media paths

There is also a rule for static paths:

acl is_short_path path_reg ^/[^/]+/s/
http-request cache-use mediacache if is_short_path

This comes from the same observation that led me to cache snac media with nginx. snac uses static media paths, and those paths often represent the kind of public, repeatable traffic that should not consume backend threads if the proxy can serve it. I call them "short", not because they are, but because the first time I saw them, I thought the 's' stood for "short", not "static". The name just stuck.

In FediMeteo this is less central than on a normal social instance, because I deliberately do not use media except for the avatar and basic static objects. Still, the rule fits the general policy: let HAProxy handle repeatable edge work, and let snac spend its threads where they are actually needed.

Vary, but not without limits

Both caches have:

process-vary on
max-secondary-entries 12

I want HAProxy to process Vary, because content negotiation is real, especially when ActivityPub is involved. But I also want variation to be bounded. If every slightly different header creates another cache entry, the cache becomes a complicated way to miss.

For media, I remove Vary before storing the response. A shared avatar does not need to vary by Accept. For ActivityPub JSON, I am more careful because the representation matters.

Again, the important thing is not the number itself. It is the decision to make variation explicit and limited.

Seeing whether it works

During rollout, I like to expose a very small diagnostic header:

http-response set-header X-Cache-Status HIT if !{ srv_id -m found }
http-response set-header X-Cache-Status MISS if { srv_id -m found }

This is intentionally simple. If HAProxy selected a backend server, I call it a miss. If no backend server was selected, the response came from cache, so I call it a hit. It is not a complete observability system, but it is enough to answer the first question I usually have after changing a cache rule.

Did this request reach snac?

A test can be as simple as:

curl -I https://it.fedimeteo.com/path/to/avatar.png
curl -I https://it.fedimeteo.com/path/to/avatar.png

The second request should be a hit.

For ActivityPub JSON, the test must use the right Accept header:

curl -I \
  -H 'Accept: application/activity+json' \
  https://it.fedimeteo.com/some/activitypub/object

And I also want to verify that cookies and authorization prevent public caching:

curl -I \
  -H 'Cookie: test=value' \
  -H 'Accept: application/activity+json' \
  https://it.fedimeteo.com/some/activitypub/object

curl -I \
  -H 'Authorization: Bearer fake' \
  -H 'Accept: application/activity+json' \
  https://it.fedimeteo.com/some/activitypub/object

A cache that works should be visible. A cache that is invisible can be correct, but it can also be silently wrong. I prefer to know.

Compression and operational paths

HAProxy also handles gzip compression:

filter compression
compression algo gzip
compression type text/css text/html text/javascript application/javascript text/plain text/xml application/json application/activity+json

This keeps another common responsibility at the edge. The country instances can stay focused on snac and the forecast data, while HAProxy deals with client-facing compression for HTML, JSON, and ActivityPub responses.

There is also a local Prometheus exporter:

frontend prometheus
  bind 127.0.0.1:8405
  mode http
  http-request use-service prometheus-exporter
  no log

And I keep internal operational paths, such as statistics and Grafana, handled before the hostname map. These are small details, but ordering matters. Special paths should be explicit and early. The hostname map is for FediMeteo routing, not for every internal tool I happen to expose behind the same proxy.

What this changes in practice

The nice thing about this configuration is that none of its parts is particularly surprising.

The map keeps hostname routing manageable. The backend definitions keep each country isolated and limited. The static homepage avoids dynamic work for something that changes once per hour. The shared avatar gives HAProxy one very hot media object to serve directly. The media cache keeps public files away from snac. The JSON microcache absorbs short ActivityPub bursts. Header cleanup prevents useless variation. Connection reuse avoids unnecessary backend connection churn.

But all of this is only a longer way of saying one thing:

fewer requests reach snac.

That is the metric I care about here.

Not because snac is slow. If anything, FediMeteo exists in its current form because snac is efficient enough to make this kind of project possible on a very small VPS. But precisely because the whole architecture is small and pleasant, I do not want to waste resources where there is no need.

This is also consistent with the rest of the project. Forecasts are serialized by scripts. Updates happen every six hours. The homepage is regenerated hourly. Countries live in separate jails. Snapshots and backups are handled outside the application. No single component tries to be the entire system.

HAProxy is just another small piece, but it sits in the right place to remove a lot of repeated work.

Caveats

This configuration is not a universal HAProxy recipe for ActivityPub services.

It matches FediMeteo as it is now: almost no media, one shared avatar, static homepage, public forecasts, many small snac instances, and ActivityPub traffic that can benefit from a short public cache when there are no cookies or authorization headers.

If I decide one day to use media in forecasts, the media cache rules will need to be reviewed. If I use different avatars for each city or country, the cache will still work, but I will lose the very nice property of one shared, always-hot avatar. If ActivityPub responses become actor-dependent, public JSON caching must be reconsidered. If one country grows a very different traffic pattern from the others, it may deserve a different limit or policy.

This is why I do not like presenting configurations as magic. A good configuration is a written form of the assumptions behind a service. When the assumptions change, the configuration must change too.

Conclusion

FediMeteo started as a small idea and became larger than I expected, but I still want it to feel small in the right ways. Small does not mean fragile. Small means understandable. It means that each part has a reason to exist, and that unnecessary work is removed before it becomes a problem.

The HAProxy layer follows this idea. It terminates TLS, routes hostnames through a map, reuses backend connections, serves the shared avatar from cache, microcaches public ActivityPub JSON, avoids authenticated and cookie-based traffic, and gives me a small diagnostic header to see what is happening.

There is no single brilliant directive here. There is only the usual work of matching infrastructure to reality.

FediMeteo publishes weather forecasts as text and emoji. The homepage is static HTML updated every hour. The accounts share the same avatar because it is enough, and because it is better for the cache. Each country has its own snac instance in its own FreeBSD jail. HAProxy stands in front of them and tries, quietly, not to bother them unless it has to.

I like this kind of infrastructure.

Not because it is invisible, but because when it works well, it leaves very little to say.

LibreNMS has been a faithful companion for years now. It quietly handles the monitoring of my servers, devices, and services without demanding much in return - exactly what you want from a tool whose job is to watch over everything else. It's a solid alternative to heavier solutions like Zabbix, and it gives you alerts, data, and graphs on virtually anything reachable over SNMP.

I usually install it on a host that is not reachable from the outside, then let it poll all the devices through a VPN: a single observation point, clean perimeter. The ability to create multiple dashboards - and to filter them by user - has also let me give clients a transparent window onto their own servers. Transparency, in my experience, is always the better long-term bet.

Together with Uptime-Kuma (and the good old Nagios/Munin pair), LibreNMS lives in a FreeBSD jail on my monitoring servers and just does its job.

This post walks through a plain installation of LibreNMS on FreeBSD: package-based, no reverse proxy, no HTTPS, no fancy hardening. The goal is to get to a working setup you can build on top of.

Assumptions

• FreeBSD 15.0-RELEASE, in a jail or on a dedicated VM/host
• nginx + php-fpm + MySQL 8.4
• LibreNMS installed from the official package — not via git clone

One note before we start: in this guide I use plain HTTP just to reach the first-time setup. If your LibreNMS instance won't stay confined to a private network or behind a VPN, configuring HTTPS is mandatory, not optional.

Installation

pkg install librenms mysql84-server python3 nginx

LibreNMS currently depends on PHP 8.4. If you want to speed PHP up, install OPcache too:

pkg install php84-opcache

MySQL

Two settings need to be in place before MySQL starts for the first time. After the first start they cannot be changed without reinitializing the data directory, so it's worth getting them right now.

cd /usr/local/etc/mysql
cp my.cnf.sample my.cnf

In the [mysqld] section, add:

innodb_file_per_table=1
lower_case_table_names=0

Now start MySQL:

service mysql-server enable
service mysql-server start

On a fresh FreeBSD install, the local root user can connect to MySQL without a password from the command line. Connect and create the database and user. I'm using password here as a placeholder - don't.

mysql

CREATE DATABASE librenms CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
CREATE USER 'librenms'@'localhost' IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON librenms.* TO 'librenms'@'localhost';
exit

php-fpm

Edit /usr/local/etc/php-fpm.d/www.conf and adjust the listen directives:

listen = /var/run/php-fpm-librenms.sock
listen.owner = www
listen.group = www
listen.mode = 0660

Then create php.ini from the production sample:

cd /usr/local/etc
cp php.ini-production php.ini

And set the timezone in php.ini:

date.timezone = Europe/Rome

nginx

Since this jail (or host) is dedicated to LibreNMS, we can rewrite the server block in /usr/local/etc/nginx/nginx.conf directly:

server {
    listen      80;
    #server_name yourServerName
    root        /usr/local/www/librenms/html;
    index       index.php;

    charset utf-8;
    gzip on;
    gzip_types text/css application/javascript text/javascript application/x-javascript image/svg+xml text/plain text/xsd text/xsl text/xml image/x-icon;

    location / {
        try_files $uri $uri/ /index.php?$query_string;
    }

    location /api/v0 {
        try_files $uri $uri/ /api_v0.php?$query_string;
    }

    location ~ \.php$ {
        fastcgi_split_path_info ^(.+\.php)(/.*)$;
        set $path_info $fastcgi_path_info;
        try_files $fastcgi_script_name =404;
        include fastcgi_params;
        fastcgi_param SERVER_SOFTWARE "";
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param PATH_INFO $path_info;
        fastcgi_index index.php;
        fastcgi_pass unix:/var/run/php-fpm-librenms.sock;
        fastcgi_buffers 256 4k;
        fastcgi_intercept_errors on;
        fastcgi_read_timeout 14400;
    }

    location ~ /\.(?!well-known).* {
        deny all;
    }
}

Now start nginx and php-fpm:

service nginx enable
service nginx start

service php_fpm enable
service php_fpm start

LibreNMS configuration

Copy the default config:

cp /usr/local/www/librenms/config.php.default /usr/local/www/librenms/config.php

Because we installed from the package, this file already has the right commands and paths for FreeBSD - no need to hunt down mtr, fping, snmpwalk and friends one by one.

Create the directory for RRD graphs and set ownership:

mkdir -p /var/db/librenms/rrd
chown -R www:www /var/db/librenms
chmod 775 /var/db/librenms/rrd

Then the .env file:

cd /usr/local/www/librenms
cp .env.example .env
chown www .env

Edit .env and set at least:

• DB_DATABASE - librenms
• DB_USERNAME - librenms
• DB_PASSWORD - the one you actually used (not password, please)

Then add this line, which tells LibreNMS we still need to run the web installer:

INSTALL=true

A note on permissions. The official LibreNMS documentation suggests chown -R www:www over the entire application tree, but on FreeBSD the package already lays down sane ownership, with storage/ and bootstrap/cache/ writable by www. There's no reason to widen the rest of the codebase. If validate.php complains later about something write-related, the first place to check is:

ls -la /usr/local/www/librenms/storage /usr/local/www/librenms/bootstrap/cache

Now generate the app key as www, since the file is owned by www:

su -m www -c "php artisan key:generate"

And tighten .env:

chmod 600 .env

Refresh the configuration cache:

su -m www -c "lnms config:clear"
su -m www -c "lnms config:cache"

Web installer

Open http://host/install and follow the steps. The validation process may fail. Refreshing the cache picks up the values written to config.php during the install:

su -m www -c "lnms config:clear"
su -m www -c "lnms config:cache"

When the web installer is done, edit .env again and remove the INSTALL=true line if it's still there. Leaving it in place re-exposes the installer to anyone who can reach the URL.

Polling service

LibreNMS needs something to actually run the polls. On FreeBSD, the package ships an rc service that runs the LibreNMS dispatcher, so there's no need to manage cron entries by hand the way most Linux guides assume.

service librenms enable
service librenms start

Validate

cd /usr/local/www/librenms
su -m www -c './validate.php'

You may see a couple of complaints right after starting the service - usually scheduler-related and self-resolving within a few minutes. Re-run validate.php once the dispatcher has had time to settle. Anything still red after that is worth investigating.

Next steps

At this point you can log into the web interface and start adding devices, configuring SNMP, and building dashboards. For that, the official LibreNMS documentation is excellent, and there's no point in me paraphrasing it here.

Yesterday morning, first thing after waking up, I checked my emails. One of them was from a client - a sharp person, but not a tech expert - forwarding a message from one of their "digital marketplaces". They claimed that during site crawling, their bot upgrades the connection to HTTP/2, and that this somehow causes issues on their end, so they were asking us to disable HTTP/2 to fix the problem.

I contacted Alex directly - the person (spoiler: not a person) who had sent the email - explaining that if their bot has trouble with HTTP/2 (which, on the contrary, provides significant benefits for the e-commerce experience in question), that's their problem, not ours, and they should fix it. Completely unprompted, I received something unexpected in reply: a guide on how to configure Apache to do what they wanted. The problem? Not only did it completely ignore my stated position, but we don't use Apache - we use nginx. And, I should add, their guide was entirely wrong. I replied pointing all of this out and finally asked to be "escalated to a human, since I was clearly talking to an AI that wasn't understanding any of my responses". The reply was blunt: "That's not possible for this type of issue. Follow our guide or we will suspend your service and your e-commerce visibility." For me, obviously, that's a hard pass. For my client, though, it's a real problem - an intelligent person who understood the situation, but still a problem to solve.

Over the past few months, I've been witnessing a dramatic increase in botnet attacks targeting some of the servers I manage, especially e-commerce ones. These aren't directed at me personally - they also hit servers I manage on behalf of clients. At first I thought they were AI scrapers, but the traffic comes from everywhere, especially from residential connections scattered around the world. I believe these are deliberate disruption campaigns, a side effect of the turbulent geopolitical climate we're living through.

On several of these e-commerce servers, we decided to implement geo-blocking, as I've described previously on this blog. Normally, once you've identified your whitelist countries and the shop isn't a global operation, everything works fine. In other cases, problems arise.

A few days ago, a partner of one of my clients - a company that provides services and needs access to some prepared XML feeds - started complaining they could no longer connect. I asked them for the IP pool they connect from, or at least the country their connections originate from. Their vague reply was: "We can't provide that information because we don't have a fixed IP or set of IPs." They completely ignored the question about the country. I pushed further, but got nowhere - different "people", giving different answers, all wildly off the mark and ignoring what I was actually asking, insisting instead that I whitelist their user agent. I explained, repeatedly, that the block is at the firewall level - meaning I never even see their user agent: if the connection is dropped, there's no handshake, no HTTP headers, nothing. It didn't matter. They kept repeating the same thing without engaging with what I wrote. Eventually they went directly to my client. I'll paste the exact text:

We're having some trouble accessing the site and downloading the XML, as they both currently require a VPN connection. To ensure our Lambda functions can run correctly, could you please:

• Remove the location-based restrictions for our access;
• Or, allow the User-Agent "REDACTED" in your firewall/server settings?

Please let us know which option works best for you.

Let's break this down:

• "Require a VPN connection" - who said anything about a VPN? Pure hallucination.
• "Remove the location-based restrictions for our access" - they never once answered: which location?
• "Allow the user agent" - I explained, multiple times, that the block is at the firewall level. The connection is dropped before any handshake occurs. There is no user agent to allow.

This morning, another client writes: "The marketing consultancy wants all the server load graphs to get an idea of where we stand." This is the second time in just a few days I've received a request like this. I send both the graphs and the full specs of the dedicated server in use - average load under 5%. The response was staggering: "The internal team, supported by the most advanced AI, believes your current setup is not adequate for the industry, load, and audience you're targeting, and recommends migrating to a cloud VPS with AT LEAST 8 GB of dedicated RAM to ensure sufficient resources, as the current ones are insufficient."

The current ones? 128 GB of RAM. Two modern CPUs. 48 cores total. If we followed their advice, the site would be down within five minutes - and that's just counting legitimate traffic. My client, unaware of the technical differences, asks me if we can implement what they're suggesting.

The shift was abrupt - not unlike when an intern arrives convinced they already know everything, often with the best of intentions: bringing fresh air into an environment that needs "modernising". But with an intern, you can talk. That same confidence often turns into curiosity, hunger to learn, real experience. I've watched eager interns grow into excellent professionals - people who eventually surpassed me in skill and success, and that felt genuinely satisfying, knowing I'd contributed, at least in part, to their growth. With AI, this is impossible. It doesn't grow, doesn't listen, doesn't update its mental model based on what you write back - and above all, it doesn't know what it doesn't know.

That's why I'd like companies to consider that AI systems are stochastic machines, not experts. They can solve some problems, but there's a limit. There will always be a limit, at least with current technology, and we can't afford to ignore it. The damage risks far outweighing the "savings" generated.

The enormous problem with my work these days is the extreme confidence that certain companies project, replacing humans - even senior ones - with AI, with no right of appeal. The result is monstrous confusion, enormous wasted time for everyone, and a widespread erosion of reliability, all papered over by the AI's unshakeable assertiveness - and by those who believe these systems are the Answer to the Ultimate Question of Life, the Universe, and Everything.

Rewarding confidence over actual competence is a bug humanity has always had. It has produced disasters throughout history, it is producing disasters now, and not only in the tech world.

So I find myself wondering: if they're so convinced that AI is better than senior professionals, why don't they replace the bosses with AI? I'm fairly confident the decisions would be considerably better - and humans would end up exactly where they should be.

Update: This post has been updated to include Docker benchmarks and a comparison of container overhead versus FreeBSD Jails and illumos Zones.

Note: Some operating systems (FreeBSD and Linux) support kernel TLS (kTLS) and the related SSL_sendfile path in nginx, which can improve HTTPS performance for static files. Since this feature is not available on all the systems included in the comparison (for example NetBSD, OpenBSD and illumos), the benchmarks were run with a common baseline configuration that does not rely on kTLS. The goal is to compare the systems under similar conditions rather than to measure OS specific optimizations.

I often get very specific infrastructure requests from clients. Most of the time it is some form of hosting. My job is usually to suggest and implement the setup that fits their goals, skills and long term plans.

If there are competent technicians on the other side, and they are willing to learn or already comfortable with Unix style systems, my first choices are usually one of the BSDs or an illumos distribution. If they need a control panel, or they already have a lot of experience with a particular stack that will clearly help them, I will happily use Linux and it usually delivers solid, reliable results.

Every now and then someone asks the question I like the least:

“But how does it perform compared to X or Y?”

I have never been a big fan of benchmarks. At best they capture a very specific workload on a very specific setup. They are almost never a perfect reflection of what will happen in the real world.

For example, I discovered that idle bhyve VMs seem to use fewer resources when the host is illumos than when the host is FreeBSD. It looks strange at first sight, but the illumos people are clearly working very hard on this, and the result is a very capable and efficient platform.

Despite my skepticism, from time to time I enjoy running some comparative tests. I already did it with Proxmox KVM versus FreeBSD bhyve, and I also compared Jails, Zones, bhyve and KVM on the same Intel N150 box. That led to the FreeBSD vs SmartOS article where I focused on CPU and memory performance on this small mini PC.

This time I wanted to do something simpler, but also closer to what I see every day: static web hosting.

Instead of synthetic CPU or I/O tests, I wanted to measure how different operating systems behave when they serve a small static site with nginx, both over HTTP and HTTPS.

This is not meant to be a super rigorous benchmark. I used the default nginx packages, almost default configuration, and did not tune any OS specific kernel settings. In my experience, careful tuning of kernel and network parameters can easily move numbers by several tens of percentage points. The problem is that very few people actually spend time chasing such optimizations. Much more often, once a limit is reached, someone yells “we need mooooar powaaaar” while the real fix would be to tune the existing stack a bit.

So the question I want to answer here is more modest and more practical:

With default nginx and a small static site, how much does the choice of host OS really matter on this Intel N150 mini PC?

Spoiler: less than people think, at least for plain HTTP. Things get more interesting once TLS enters the picture.

Disclaimer
These benchmarks are a snapshot of my specific hardware, network and configuration. They are useful to compare relative behavior on this setup. They are not a universal ranking of operating systems. Different CPUs, NICs, crypto extensions, kernel versions or nginx builds can completely change the picture.

Test setup

The hardware is the same Intel N150 mini PC I used in my previous tests: a small, low power box that still has enough cores to be interesting for lab and small production workloads.

On it, I installed several operating systems and environments, always on the bare metal, not nested inside each other. On each OS I installed nginx from the official packages.

Software under test

On the host:

SmartOS, with:
- a Debian 12 LX zone
- an Alpine Linux 3.22 LX zone
- a native SmartOS zone

FreeBSD 14.3-RELEASE:
- nginx running inside a native jail

OpenBSD 7.8:
- nginx on the host

NetBSD 10.1:
- nginx on the host

Debian 13.2:
- nginx on the host

Alpine Linux 3.22:
- nginx on the host
- Docker: Debian 13 container running on the Alpine host (ports mapped)

I also tried to include DragonFlyBSD, but the NIC in this box is not supported. Using a different NIC just for one OS would have made the comparison meaningless, so I excluded it.

nginx configuration

In all environments:

• nginx was installed from the system packages
• worker_processes was set to auto
• the web root contained the same static content

The important part is that I used exactly the same nginx.conf file for all operating systems and all combinations in this article. I copied the same configuration file verbatim to every host, jail and zone. The only changes were the IP address and file paths where needed, for example for the TLS certificate and key.

The static content was a default build of the example site generated by BSSG, my Bash static site generator. The web root was the same logical structure on every OS and container type.

There is no OS specific tuning in the configuration and no kernel level tweaks. This is very close to a “package install plus minimal config” situation.

TLS configuration

For HTTPS I used a very simple configuration, identical on every host.

Self signed certificate created with:

openssl req -x509 -newkey rsa:4096 -nodes -keyout server.key -out server.crt -days 365 -subj "/CN=localhost"  

Example nginx server block for HTTPS (simplified):

server {  
listen 443 ssl http2;  
listen [::]:443 ssl http2;  

server_name _;  

ssl_certificate /etc/nginx/ssl/server.crt;  
ssl_certificate_key /etc/nginx/ssl/server.key;  

root /var/www/html;  
index index.html index.htm;  

location / {  
try_files $uri $uri/ =404;  
}  
}  

The HTTP virtual host is also the same everywhere, with the root pointing to the BSSG example site.

Load generator

The tests were run from my workstation on the same LAN:

• client host: a mini PC machine connected at 2.5 Gbit/s
• switch: 2.5 Gbit/s
• test tool: wrk

For each target host I ran:

• wrk -t4 -c50 -d10s http://IP
• wrk -t4 -c10 -d10s http://IP
• wrk -t4 -c50 -d10s https://IP
• wrk -t4 -c10 -d10s https://IP

Each scenario was executed multiple times to reduce noise; the numbers below are medians (or very close to them) from the runs.

The contenders

To keep things readable, I will refer to each setup as follows:

• SmartOS Debian LX → SmartOS host, Debian 12 LX zone
• SmartOS Alpine LX → SmartOS host, Alpine 3.22 LX zone
• SmartOS Native → SmartOS host, native zone
• FreeBSD Jail → FreeBSD 14.3-RELEASE, nginx in a jail
• OpenBSD Host → OpenBSD 7.8, nginx on the host
• NetBSD Host → NetBSD 10.1, nginx on the host
• Debian Host → Debian 13.2, nginx on the host
• Alpine Host → Alpine 3.22, nginx on the host
• Docker Container → Alpine host, Debian 13 Docker container

Everything uses the same nginx configuration file and the same static site.

Static HTTP results

Let us start with plain HTTP, since this removes TLS from the picture and focuses on the kernel, network stack and nginx itself.

HTTP, 4 threads, 50 concurrent connections

Approximate median wrk results:

Two things stand out:

1. All the native or jail/container setups on the hosts that are not LX zones cluster around 63 to 64k requests per second.
2. The two SmartOS LX zones sit slightly lower, in the 46 to 49k range, which is still very respectable for this hardware.

In other words, as long as you are on the host or in something very close to it (FreeBSD jail, SmartOS native zone, NetBSD, OpenBSD, Linux on bare metal), static HTTP on nginx will happily max out around 64k requests per second with this small Intel N150 CPU.

The Debian and Alpine LX zones on SmartOS are a bit slower, but not dramatically so. They still deliver close to 50k requests per second and, in a real world scenario, you would probably saturate the network or the client long before hitting those numbers.

HTTP, 4 threads, 10 concurrent connections

With fewer concurrent connections, absolute throughput drops, but the relative picture is similar:

• SmartOS Native around 44k
• NetBSD and Alpine Host around 34 to 35k
• FreeBSD, Debian, OpenBSD around 31 to 33k
• The Docker Container sits slightly lower at ~30.2k req/s, showing a small overhead from the networking layer
• The SmartOS LX zones sit slightly below, around 35 to 37k req/s

The important conclusion is simple:

For plain HTTP static hosting, once nginx is installed and correctly configured, the choice between these operating systems makes very little difference on this hardware. Zones and jails add negligible overhead, LX zones add a small one.

If you are only serving static content over HTTP, your choice of OS should be driven by other factors: ecosystem, tooling, update strategy, your own expertise and preference.

Static HTTPS results

TLS is where things start to diverge more clearly and where CPU utilization becomes interesting.

HTTPS, 4 threads, 50 concurrent connections

Approximate medians:

These numbers tell a more nuanced story.

1. FreeBSD, Debian and Alpine on bare metal form a “fast TLS” group.
   All three sit around 62 to 63k requests per second with 50 concurrent HTTPS connections.

2. FreeBSD does this while using significantly less CPU.
   During the HTTPS tests with 50 connections, the FreeBSD host still had around 60% CPU idle. It is the platform that handled TLS load most comfortably in terms of CPU headroom.

3. Debian and Alpine are close in throughput, but push the CPU harder.
   Debian still had some idle time left, Alpine even less. In practice, all three are excellent here, but FreeBSD gives you more room before you hit the wall.

4. SmartOS, NetBSD and OpenBSD form a “good but heavier” TLS group.
   Their HTTPS throughput is in the 40 to 52k req/s range and they reach full CPU usage at 50 concurrent connections. OpenBSD and NetBSD stabilize around 39 to 40k req/s. SmartOS native and the Debian LX zone manage slightly better (around 51 to 53k) but still with the CPU pegged.

HTTPS, 4 threads, 10 concurrent connections

With lower concurrency:

• FreeBSD, Debian and Alpine still sit in roughly the 29 to 31k req/s range
• SmartOS Native and LX zones are in the mid to high 30k range
• The Docker Container drops slightly to ~27.8k req/s
• NetBSD and OpenBSD sit around 26 to 27k req/s

The relative pattern is the same: for this TLS workload, FreeBSD and modern Linux distributions on bare metal appear to make better use of the cryptographic capabilities of the CPU, delivering higher throughput or more headroom or both.

What TLS seems to highlight

The HTTPS tests point to something that is not about nginx itself, but about the TLS stack and how well it can exploit the hardware.

On this Intel N150, my feeling is:

• FreeBSD, with the userland and crypto stack I am running, is very efficient at TLS here. It delivers the highest throughput while keeping plenty of CPU in reserve.
• Debian and Alpine, with their recent kernels and libraries, are also strong performers, close to FreeBSD in throughput, but with less idle CPU.
• NetBSD, OpenBSD and SmartOS (native and LX) are still perfectly capable of serving a lot of HTTPS traffic, but they have to work harder to keep up and they hit 100% CPU much earlier.

This matches what I see in day to day operations: TLS performance is often less about “nginx vs something else” and more about the combination of:

• the TLS library version and configuration
• how well the OS uses the CPU crypto instructions
• kernel level details in the network and crypto paths

I suspect the differences here are mostly due to how each system combines its TLS stack (OpenSSL, LibreSSL and friends), its kernel and its hardware acceleration support. It would take a deeper dive into profiling and configuration knobs to attribute the gaps precisely.

In any case, on this specific mini PC, if I had to pick a platform to handle a large amount of HTTPS static traffic, FreeBSD, Debian and Alpine would be my first candidates, in that order.

Zones, jails, containers and Docker: overhead in practice

Another interesting part of the story is the overhead introduced by different isolation technologies.

From these tests and the previous virtualization article on the same N150 machine, the picture is consistent:

• FreeBSD jails behave almost like bare metal and are significantly more efficient than Docker.
  For both HTTP and HTTPS, running nginx in a jail on FreeBSD 14.3-RELEASE produces numbers practically identical to native hosts.
  The contrast with Docker is striking: while the Docker container required 100% CPU to reach peak for the HTTP and HTTPS throughput, the FreeBSD jail delivered the same speed with ~60% of the CPU sitting idle. In terms of performance cost per request, Jails are drastically cheaper.

• SmartOS native zones are also very close to the metal.
  Static HTTP performance reaches the same 64k req/s region and HTTPS is only slightly behind the "fast TLS" group, although with higher CPU usage.

• SmartOS LX zones introduce a noticeable but modest overhead.
  Both Debian and Alpine LX zones on SmartOS perform slightly worse than the native zone or FreeBSD jails. For static HTTP they are still very fast. For HTTPS the Debian LX zone remains competitive but costs more CPU, while the Alpine LX zone is slower.

• Docker on Linux performs efficiently but eats the margins. I ran an additional test using a Debian 13 Docker container running on the Alpine Linux host. At peak load (50 connections), the throughput was impressive and virtually identical to bare metal: ~63.7k req/s for HTTP and ~62.7k req/s for HTTPS. However, there is a clear cost. First, while the bare metal host maintained a small CPU buffer (~7% idle) during the HTTPS test, Docker saturated the CPU to 100%. Second, at lower concurrency (10 connections), the overhead became visible. The Docker container scored ~30.2k req/s for HTTP and ~27.8k req/s for HTTPS, slightly trailing the ~31-34k and ~29-31k range of the bare metal counterparts. The abstraction layers (NAT, bridging, namespaces) are extremely efficient, but they are not completely free.

This leads to a clear conclusion on efficiency: FreeBSD Jails provide the highest throughput with the lowest CPU cost. LX zones and Docker containers can match the speed (or come close), but they burn significantly more CPU cycles to do so.

What this means for real workloads

It is easy to get lost in tables and percentages, so let us go back to the initial question.

A client wants static hosting.
Does the choice between FreeBSD, SmartOS, NetBSD or Linux matter in terms of performance?

For plain HTTP on this hardware, with nginx and the same configuration:

• Not really.
  All the native hosts and FreeBSD jails deliver roughly the same maximum throughput, in the 63 to 64k req/s range. SmartOS LX zones are slightly slower but still strong.

For HTTPS:

• Yes, it starts to matter a bit more.
• FreeBSD stands out for how relaxed the CPU is under high TLS load.
• Debian and Alpine are very close in throughput, with more CPU used but still with some headroom.
• SmartOS, NetBSD and OpenBSD can still push a lot of HTTPS traffic, but they reach 100% CPU earlier and stabilize at lower request rates.

Does this mean you should always choose FreeBSD or Debian or Alpine for static HTTPS hosting?

Not necessarily.

In real deployments, the bottleneck is rarely the TLS performance of a single node serving a small static site. Network throughput, storage, logging, reverse proxies, CDNs and application layers all play a role.

However, knowing that FreeBSD and current Linux distributions can squeeze more out of a small CPU under TLS is useful when you are:

• sizing hardware for small VPS nodes that must serve many HTTPS requests
• planning to consolidate multiple services on a low power box
• deciding whether you can afford to keep some CPU aside for other tasks (cache, background jobs, monitoring, and so on)

As always, the right answer depends on the complete picture: your skills, your tooling, your backups, your monitoring, the rest of your stack, and your tolerance for troubleshooting when things go sideways.

Final thoughts

From these small tests, my main takeaways are:

1. Static HTTP is basically solved on all these platforms.
   On a modest Intel N150, every system tested can push around 64k static HTTP requests per second with nginx set to almost default settings. For many use cases, that is already more than enough.

2. TLS performance is where the OS and crypto stack start to matter.
   FreeBSD, Debian and Alpine squeeze more HTTPS requests out of the N150, and FreeBSD in particular does it with a surprising amount of idle CPU left. NetBSD, OpenBSD and SmartOS need more CPU to reach similar speeds and stabilize at lower throughput once the CPU is saturated.

3. Jails and native zones are essentially free, LX zones cost a bit more.
   FreeBSD jails and SmartOS native zones show very little overhead for this workload. SmartOS LX zones are still perfectly usable, but if you are chasing every last request per second you will see the cost of the translation layer.

4. Benchmarks are only part of the story.
   If your team knows OpenBSD inside out and has tooling, scripts and workflows built around it, you might happily accept using more CPU on TLS in exchange for security features, simplicity and familiarity. The same goes for NetBSD or SmartOS in environments where their specific strengths shine.

I will not choose an operating system for a client just because a benchmark looks nicer. These numbers are one of the many inputs I consider. What matters most is always the combination of reliability, security, maintainability and the human beings who will have to operate the
system at three in the morning when something goes wrong.

Still, it is nice to know that if you put a tiny Intel N150 in front of a static site and you pick FreeBSD or a modern Linux distribution for HTTPS, you are giving that little CPU a fair chance to shine.

Mastodon 4.5.0 is here, and with it come some interesting changes that, in my opinion, might encourage more people to consider it for self-hosting their Fediverse community.

While it may not be as lightweight and simple as other solutions (like snac or GoToSocial or many others), I believe it remains one of the best platforms for managing a medium-sized Fediverse community, thanks in part to the direct feedback that many admins have provided to the developers.

I have previously written about how to install Mastodon in a FreeBSD jail and how to modify its character and poll limits.

One of the most critical initial decisions (which can be changed later, but with extra work) is where to store your media files. Mastodon downloads and re-processes all media it encounters from other instances for three main reasons:

• Local Caching: Your users connect to your media server, reducing the load on the original instance.
• Security: Re-processing media helps to remove any potential "impurities" before they reach the user's device.
• Privacy: It prevents disclosing your users' IP addresses to other instances. A user will only connect to their own instance to fetch all data, including remote content.

At least initially, media files will be the largest part of your instance's storage footprint. It is therefore essential to plan where to store them and to add a regular cleanup script; otherwise, their growth will be exponential.

Mastodon supports uploading media to external S3-compatible solutions, and many admins use the usual commercial providers, paying for data uploads and transfers.

I am a firm believer in "Own Your Data", so I have always used my own self-hosted S3 servers. I initially started with Minio, but over time, I realized that, by design, it doesn't perform well with a multitude of small files (performance degrades). After running some tests, I decided to switch to SeaweedFS.

SeaweedFS "is a fast distributed storage system for blobs, objects, files, and data lake, for billions of files! Blob store has O(1) disk seek..." - this, combined with the fact that it is a mature and proven piece of software, was enough for me to give it a try. The result? Excellent. The I/O and CPU load on my media server dropped drastically, making SeaweedFS an incredibly suitable solution. Furthermore, some of its features (like the ability to run a filer.sync) allow for efficient and fast replication to other storage, another host, or... anything else.

SeaweedFS works perfectly with Mastodon, and I will explain the steps to get it into production.

I will install SeaweedFS in a dedicated jail and use a dedicated subdomain. This ensures that the media server can be moved to another host at any time without reconfiguring everything or changing domains. SeaweedFS has its own FreeBSD package, installable via pkg, or can be downloaded directly from the project's website.

In either case, I will describe a "test" setup - which can also be used in production without issues. However, I highly recommend diving deeper into the tool, as it is incredibly powerful and flexible and can solve many more problems than one might imagine.

Setting up the SeaweedFS Jail

First, let's create a dedicated jail with BastilleBSD:

bastille create media 14.3-RELEASE 10.0.0.66 bastille0

Now, let's enter the jail and install SeaweedFS (and tmux, which can be useful):

bastille console media
pkg install -y tmux seaweedfs

I suggest launching SeaweedFS in a tmux session so you can monitor its output. Later, you should configure an automatic startup method, such as using the included rc.d file or any other method you prefer.

Create a directory for the data and start SeaweedFS as the "seaweedfs" user:

mkdir -p /seaweedfs/data
chown -R seaweedfs /seaweedfs
su -m seaweedfs
cd /seaweedfs/
/usr/local/bin/weed server -dir /seaweedfs/data -s3

At this point, SeaweedFS will start and create everything it needs to function, including the S3 server.

Configuring Buckets and Users

Now, let's open the weed shell to create the necessary bucket and users:

weed shell
s3.bucket.create -name mastomedia

Still in the weed shell, create a user for Mastodon and grant read permissions for unauthenticated users (which is necessary to serve media to the world):

s3.configure -access_key=mastomedia -secret_key=CHANGEME -buckets=mastomedia -user=mastodon -actions=Read,Write,List,Tagging,Admin -apply
s3.configure -buckets=mastomedia -user=anonymous -actions=Read -apply
s3.configure -buckets=mastomedia -actions=Read -apply

Security Tip: For the -secret_key, avoid using a simple password. You can generate a strong, random key directly from your shell with a command like openssl rand -base64 32.

Done. SeaweedFS is now ready to receive (and serve) media. The next step is to set up a reverse proxy to serve everything over HTTPS. My preferred approach is to configure the system as if it were external, even if the services are in adjacent jails. This might use slightly more resources, but the time and trouble it saves in the future are well worth it.

Nginx Reverse Proxy Configuration

The reverse proxy can be configured something like this:

[...]

server {
   server_name  media.mastodon.example.com;

   ignore_invalid_headers off;
   client_max_body_size 0; # Allow large file uploads without Nginx limits

   location / {
      proxy_set_header Host $http_host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;

      proxy_connect_timeout 300;
      proxy_http_version 1.1;
      proxy_set_header Connection "";
      chunked_transfer_encoding off;

      expires 1y;
      add_header Cache-Control public;

      add_header X-Cache-Status $upstream_cache_status;
      add_header X-Content-Type-Options nosniff;

      proxy_pass http://10.0.0.66:8333;
   }

# ... other server configurations like SSL ...

}

Mastodon Configuration

Now let's configure Mastodon. If you are running the setup wizard for the first time, here is a summary of the options:

[...]
Do you want to store uploaded files on the cloud? yes
Provider Minio
Minio endpoint URL: https://media.mastodon.example.com
Minio bucket name: mastomedia
Minio access key: mastomedia
Minio secret key: CHANGEME
Do you want to access the uploaded files from your own domain? Yes
Domain for uploaded files: media.mastodon.example.com

If Mastodon is already active, or once the setup is complete, the options in your .env.prod file should be modified to be consistent with what SeaweedFS expects:

S3_ENABLED=true
S3_PROTOCOL=https
S3_REGION=us-east-1
S3_ENDPOINT=https://media.mastodon.example.com
S3_HOSTNAME=media.mastodon.example.com
S3_BUCKET=mastomedia
AWS_ACCESS_KEY_ID=mastomedia
AWS_SECRET_ACCESS_KEY=CHANGEME
S3_FORCE_SINGLE_REQUEST=true
# remove the S3_ALIAS_HOST if it is set

IMPORTANT NOTE: If both services are in jails on the same host (i.e., SeaweedFS is on the same host as Mastodon), you should ensure that the Mastodon jail can reach the SeaweedFS jail through the reverse proxy and not via the external IP. To do this, add the following line to the /etc/hosts file of the Mastodon jail:

10.0.0.1        media.mastodon.example.com

In this example, the reverse proxy is at 10.0.0.1. If you are not using a separate reverse proxy but are exposing Nginx directly from the jail (as described in my Mastodon installation article), use the IP of the Mastodon jail itself instead (e.g., 10.0.0.42).

With this setup, Mastodon will be able to upload media to the SeaweedFS server and generate the correct links for other instances, public visitors, and users of your own instance.

Have fun with SeaweedFS!

Disclaimer
These benchmarks were performed on my specific hardware and tuned for the workloads I expect to run.
They should not be taken as absolute or universally applicable results.
Different CPUs, storage, networking setups, or workload profiles could produce very different outcomes.
What I’m sharing here is a faithful snapshot of my test environment and use case - a guidepost, not a final verdict.

Years ago, I installed a PCEngines APU at a client's site. It dutifully ran Proxmox with a few small VMs inside. It wasn't a speed demon, but it got the job done. Tasked with running in a closed, uncooled, and unsupervised server closet, it soldiered on for about seven years.

Then, while I was at BSDCan, I got the call. A series of power outages and surges had finally taken their toll, and the APU was dead. It was probably just the power supply, but given its age, we decided it was time for a replacement. I set up a remote bypass to keep them running, but I knew I'd need to install something more powerful soon.

I ordered a modern MiniPC-based on the low-power Intel Processor N150 platform, but with 16GB of RAM and more than enough performance to serve as a decent workstation. I have a similar one in my office running openSUSE Tumbleweed, and it works beautifully.

This time, however, I decided to replace Proxmox with a different virtualization system. This decision wasn't made in a vacuum. In the past, I've put bhyve head-to-head with Proxmox, and my findings were clear: bhyve on FreeBSD is an extremely efficient hypervisor, often outperforming KVM on Proxmox in my tests.

This positive experience is what made FreeBSD with bhyve a top contender. The other path was a KVM-style approach (which would require fewer changes to the VMs), where my options would be NetBSD or an illumos-based OS like SmartOS. Since I had the new hardware on hand, I decided to run some tests to see how these different technologies stacked up against each other, and against the bare metal itself.

The Lineup: What I Put on the Test Bench

My goal was to test every reasonable option on this Intel N150 hardware. The final lineup covered the entire spectrum:

• The Baseline:
  • FreeBSD 14.3-RELEASE Bare Metal: The ground truth for performance on this hardware.
• OS-Level Virtualization (Containers):
  • SmartOS Native Zone: The baseline native container on SmartOS.
  • SmartOS LX Zone: Running Ubuntu 24.04 and Alpine Linux.
  • FreeBSD Native Jail: The baseline native container on FreeBSD.
  • FreeBSD Jail with Linux: A jail running a Ubuntu 22.04 userland.
• Full Hardware Virtualization (HVM):
  • SmartOS bhyve Zone: A FreeBSD guest inside the bhyve hypervisor on a SmartOS host.
  • SmartOS KVM Zone: A FreeBSD guest inside the KVM hypervisor on a SmartOS host.
  • FreeBSD bhyve VM: A FreeBSD guest inside the bhyve hypervisor on a FreeBSD host.

The Benchmark: My sysbench Commands

To keep the comparison fair and simple, I used two core sysbench commands. To ensure consistency, I even compiled sysbench from scratch on the SmartOS native zone to match the versions and compile options on the other systems as closely as possible.

The commands I used in each environment were:

• For CPU performance: sysbench --test=cpu --cpu-max-prime=20000 run
• For memory performance: sysbench --test=memory run

First Look: CPU and Memory on the Intel N150

My initial tests on the Intel N150 hardware immediately revealed some interesting trends. The sysbench CPU results from any native FreeBSD environment (bare metal or jail) were on a completely different scale from the Linux and SmartOS guests, making a direct comparison meaningless.

However, by excluding the incompatible FreeBSD-native results, we get a very clear picture of the overhead between the various container technologies.

Valid CPU Performance Comparison (Single Thread, Intel N150)

The takeaway here was clear: for CPU work, the overhead from these containers is basically a rounding error. For CPU-bound tasks, neither SmartOS Zones nor FreeBSD Jails will be a bottleneck.

The memory results, which were consistent across all platforms, were far more revealing.

Overall Memory Performance Comparison (Intel Processor N150)

These initial numbers led to a few conclusions: a virtual layer could be a performance boost, the userland matters, and bhyve clearly outclassed the legacy KVM on SmartOS. However, one result was nagging at me: the performance gap between FreeBSD bare metal (4005.08 MiB/sec) and a native bhyve VM (3636.01 MiB/sec) was about 9%. This was a larger drop than I expected. It prompted a new question: was this overhead inherent to bhyve, or was it a quirk of the new N150 hardware?

Going deeper: Testing on an Intel i7-7500U

To see if more mature, better-supported hardware would tell a different story, I replicated the FreeBSD tests on an older Qotom Mini-PC powered by an Intel i7-7500U. The results were illuminating and dramatically changed the narrative.

CPU Performance Comparison (Intel i7-7500U)

Once again, the CPU tests produced strange results. The native FreeBSD environments all reported incredibly high numbers in the millions of events/sec, while the Ubuntu Linuxulator jail's result was on a completely different, incompatible scale. Frankly, given the massive discrepancy between FreeBSD-native and Linux-based environments, I'm unsure that the sysbench CPU figures can be considered totally reliable in absolute terms.

However, what is useful is comparing the native FreeBSD results against each other. This tells us about relative overhead.

Even if we're skeptical of the absolute numbers, the relative comparison is crystal clear: the CPU overhead of bhyve is less than half a percent. This is the key takeaway.

Memory Performance Comparison (Intel i7-7500U)

The memory benchmarks, in contrast, were consistent and highly informative.

This is where the real story is. The memory performance of a bhyve VM was a mere 0.52% slower than bare metal. This is the kind of near-native performance one hopes for from a top-tier hypervisor and stands in stark contrast to the 9% drop seen on the newer N150.

Breaking Down the Results: What I Learned From Both Tests

This comprehensive two-platform analysis paints a much clearer picture.

1. Hardware Really Matters Performance is not an absolute. The difference between the two platforms was stark: on the mature i7-7500U, bhyve’s overhead was less than 1%, while on the newer, budget N150, it was a more significant 9%. This suggests the performance dip is likely due to missing optimizations for that specific CPU architecture, rather than a fundamental flaw in bhyve itself.

2. bhyve's True Potential is Near-Native Speed The i7 tests prove that bhyve is an exceptionally efficient hypervisor on well-supported hardware. The relative CPU overhead was a negligible -0.48%, and more importantly, the reliable memory benchmarks showed a performance drop of just 0.52% compared to bare metal. This is the gold standard for virtualization.

3. FreeBSD Jails are Feather-Light On both platforms, native FreeBSD jails demonstrated almost zero performance overhead. On the i7, both CPU and memory performance were virtually identical to bare metal (a 0.05% difference). The N150 CPU tests further showed that FreeBSD's container implementation is so efficient that running a Linux userland inside a jail delivered the best CPU scores of the entire lineup.

4. SmartOS Zones Are Also Extremely Efficient Just like Jails, SmartOS's native Zones proved to be remarkably lightweight. The N150 CPU tests confirm this, showing that native and LX zones have virtually identical, top-tier performance. On the memory front, the native Zone delivered performance over 13% faster than the FreeBSD bare-metal baseline, pointing to the high efficiency of the illumos kernel.

5. The Linux Userland Excels at Throughput A clear pattern emerged on both testbeds: the Ubuntu userland consistently delivered excellent benchmark results. On the CPU front, Ubuntu on both FreeBSD and SmartOS delivered the highest, and nearly identical, performance scores on the N150. For memory, the story was even more dramatic: the Ubuntu LX Zone on SmartOS was the top performer, beating bare-metal FreeBSD by nearly 25%, while the Ubuntu jail on the i7 also surpassed its host by over 7%.

Final Thoughts: The Verdict for My Client's New Server

So, what's the bottom line for my client's new MiniPC? This benchmarking journey has made the path forward much clearer.

At the beginning of this process, my main question was whether to stick with a KVM-based setup or make the switch to bhyve. The performance data answers that decisively. The legacy KVM on SmartOS showed a crippling performance penalty, making it a non-starter. Given that, the extra effort to migrate the existing VMs to a bhyve-compatible format is absolutely worth it. The performance gain is just too significant to ignore.

The final question, then, is which host OS to use for bhyve: SmartOS or FreeBSD? This is a much tougher call, as both platforms demonstrated incredible strengths.

SmartOS, powered by the illumos kernel, was a true surprise. It delivered astonishing performance on the target N150 hardware. Its key advantage is the raw speed of its containerization for both CPU and memory tasks. The Ubuntu LX Zone not only ran flawlessly but delivered top-tier CPU scores and outperformed the bare-metal FreeBSD baseline in memory by a massive 25% margin. This points to a highly efficient kernel and offers the tantalizing prospect of running ultra-fast Linux containers alongside performant bhyve VMs on the same host.

On the other hand, FreeBSD proved its mastery of bhyve virtualization. The tests on the i7 hardware showed its implementation to be the gold standard, offering virtually zero performance overhead for full hardware virtualization. Its native Jails are equally efficient, and its Linux compatibility layer is so effective that an Ubuntu jail delivered the fastest CPU performance of all containers tested on FreeBSD. For workloads that must live in a full VM, FreeBSD offers the most performant and native bhyve experience, with the reasonable expectation that its support for newer hardware like the N150 will only improve over time.

Ultimately, the choice comes down to the primary workload. It's a decision between the raw container speed and Linux flexibility of SmartOS versus the pure, uncompromising HVM performance of FreeBSD.

But one thing is certain: thanks to this deep dive, the path forward is much clearer, and it's paved by bhyve.

New Article Published

I'm excited to announce that I have published a new, in-depth article on the BSD Cafe Journal: "WordPress on FreeBSD with BastilleBSD: A Secure Alternative to Linux/Docker".

This piece explores how to create a robust and secure WordPress installation on FreeBSD using BastilleBSD, leveraging the power and isolation of FreeBSD jails as a compelling alternative to the more common Linux and Docker stack.

Future Technical Content

I'm excited to announce that I'm expanding my writing to a new platform! From now on, some of my more technical, long-form articles and tutorials will be published on The BSD Cafe Journal, a fantastic hub for BSD-related content that I'm happy to now contribute to.

This new collaboration complements the work I do here. My personal blog will continue to be my home base, and you won't miss a thing! I'll still be posting my own articles and announcements right here, and I'll always include a direct link to any new content I publish elsewhere. This space will remain as active as ever.

Thank you for reading

FreeBSD is an extremely powerful operating system. The ability to isolate services in jails and, thanks to ZFS, the simplicity with which you can create snapshots (both local and remote) make it a perfect system for increasing peace of mind, especially when running many workloads.

Many providers, blinded by the success and large numbers achieved by Linux distributions, have decided to no longer support FreeBSD in their installers. While understandable (they might not have staff experienced with systems other than Linux), this can cause problems for those who want to try using something different. And yes, it makes perfect sense, even just to avoid IT monocultures, which are extremely harmful even in the medium term.

There's an extremely powerful tool that, in my opinion, deserves much more attention than it gets. The tool is called mfsBSD. Using the author's words: "This is a set of scripts that generates a bootable image (and/or ISO file), that creates a working minimal installation of FreeBSD (mfsBSD) or Linux (mfslinux). It is completely loaded into memory."

mfsBSD works intelligently: it can be launched both via UEFI and via "traditional" boot since it has both boot modes enabled. It gets an IP address via DHCP (an operation that works with most providers) and opens an SSH server (with a preset password - so it's advisable to connect immediately and change it, to prevent someone else from doing it for you).

This means that mfsBSD can be used both when you have a console available and, in many cases, without a console, since you'll just need to connect via SSH and start with the traditional installation.

To install FreeBSD using mfsBSD, you just need to follow some very simple steps: all providers, in fact, offer the ability to boot in Linux "rescue mode", generally based on a sufficiently recent version. Set your server (whether physical or VPS, it doesn't matter) in rescue mode and reboot. Once active, connect via SSH (or open a console) to the server in rescue mode.

Now it's sufficient to download the mfsBSD image (for example, from here: https://mfsbsd.vx.sk/files/images/ - in my case, I usually choose the normal image, which at the time of writing this post is https://mfsbsd.vx.sk/files/images/14/amd64/mfsbsd-14.2-RELEASE-amd64.img). At this point, it needs to be written with dd directly to the server's disk (or, if in doubt, disks). For example:

dd if=mfsbsd-14.2-RELEASE-amd64.img of=/dev/sda bs=5M conv=sync

In this case I wrote "sda", but if you had one or more NVMe drives, the correct device would be /dev/nvme0n1 for the first disk, /dev/nvme1n1 for the second, etc.

Reboot. If you have a console, you'll see mfsBSD boot up, enable the SSH server, and position itself at the login. Otherwise, ping the server's IP until it starts responding. At that point, it's sufficient to connect via SSH as the root user.

The root password is mfsroot

As the first thing, change the root password with the passwd command. This is to prevent someone from entering and compromising the machine during the installation time.

Now launch the bsdinstall command and proceed with the normal FreeBSD installation (also setting up mirrors, RAID, etc., if desired), keeping in mind that mfsBSD is running in RAM, so you can overwrite the disk it was installed on without problems.

After the reboot, you'll have your FreeBSD system installed and running.

Update, September 26, 2024: WordPress.org has banned websites hosted on WP Engine from accessing its resources. As someone put it on X, this is Matt Mullenweg dropping a giant turd into the laps of millions of WordPress users. I can't recommend this software any more in good conscience.

The Matt Mullenweg v. WP Engine dispute seems to be escalating, which is a bit of a surprise because it was so ill-founded to begin with. Yet the escalation has also been exponential.

Mullenweg published his post disparaging WP Engine on the WordPress.org site (from where you can download the open source WordPress CMS) on September 21.

Matt Mullenweg v. WP Engine

Automattic CEO and WordPress co-developer Matt Mullenweg published a post on September 21 calling WP Engine a “cancer to WordPress”. For the uninitiated: WP Engine is an independent company that provides managed hosting for WordPress sites; WordPress.com is owned by Automattic and it leads the development of WordPress.org.

Close ReadVasudevan Mukunth

On September 23, WP Engine said it had sent WordPress.com parent company and WordPress lead developer Automattic, whose CEO is Mullenweg, a cease and desist letter. Excerpt:

Stunningly, Automattic’s CEO Matthew Mullenweg threatened that if WP Engine did not agree to pay Automattic – his for-profit entity – a very large sum of money before his September 20th keynote address at the WordCamp US Convention, he was going to embark on a self-described “scorched earth nuclear approach” toward WP Engine within the WordPress community and beyond. When his outrageous financial demands were not met, Mr. Mullenweg carried out his threats by making repeated false claims disparaging WP Engine to its employees, its customers,and the world. Mr. Mullenweg has carried out this wrongful campaign against WP Engine in multiple outlets, including via his keynote address, across several public platforms like X, YouTube, and even on the Wordpress.org site, and through the WordPress Admin panel for all WordPress users, including directly targeting WP Engine customers in their own private WordPress instances used to run their online businesses.

💥

Later on the same day, Automattic sent a cease and desist letter of its own to WP Engine. Excerpt:

As you know, our Client owns all intellectual property rights globally in and to the world-famous WOOCOMMERCE and WOO trademarks; and the exclusive commercial rights from the WordPress Foundation to use, enforce, and sublicense the world-famous WORDPRESS trademark, among others, and all other associated intellectual property rights.We are writing about WP Engine’s web hosting and related services that improperly use our Client’s WORDPRESS and WOOCOMMERCE trademarks in their marketing.We understand that our Client has contacted you about securing a proper license to use its trademarks, yet no such agreement has been reached. As such, your blatant and widespread unlicensed use of our Client’s trademarks has infringed our Client’s rights and confused consumers into believing, falsely, that WP Engine is authorized, endorsed, or sponsored by, or otherwise affiliated or associated with, our Client. WP Engine’s unauthorized use of our Client’s trademarks also dilutes their rights, tarnishes their reputation, and otherwise harms the goodwill they have established in their famous and well-known trademarks, and has enabled WP Engine to unfairly compete with our Client, leading to WP Engine’s unjust enrichment.

Now it's a trademark dispute. Automattic is alleging people at large are confusing WP Engine with WordPress itself and that that's leading to loss of revenue for Automattic. Hang on to this thought while we move on to the next detail. At 10.34 pm IST on September 4, Mullenweg shared this tidbit in a Reddit comment:

[WP Engine] had the option to license the WordPress trademark for 8% of their revenue, which could be delivered either as payments, people (Five for the Future .org commitments), or any combination of the above.

Put all these details together and we understand Mullenweg is alleging via Automattic that people are confusing WP Engine with WordPress itself to Automattic's detriment, that WP Engine has wrongfully used the WordPress trademark, that what WP Engine is selling isn't WordPress but something it has reportedly "butchered", and that WP Engine's enrichment is unjust.

I think it's starting to stink for Mullenweg. As detailed in the previous post, WP Engine didn't "butcher" WordPress. In fact they didn't change anything about WordPress's core composition. They turned off a setting, didn't hide it, and offered a way to get around it by other means. WordPress is open source software provided under a GPL license, which means others are allowed to modify it (and subsequently avail it under the same license). So even if WP Engine modified WordPress — which it didn't — it would've been operating within its rights.

An opportunity to understand the GPL license

Featured image: Matt Mullenweg, 2009. Credit: loiclemeur/Flickr, CC BY 2.0. Every December, I wander over to ma.tt, the blog of WordPress founder Matt Mullenweg, to check what he’s saying about how the CMS will be shaping up in the next year. Despite my cribbings as well

Close ReadVM

Second, WP Engine was founded in 2010. Why is Automattic alleging a trademark violation after 14 years of being okay with it? Even if consumers are currently confusing WP Engine with WordPress itself — which I doubt — that Automattic didn't pursue a legal dispute in all this time is very fishy. It also creates new uncertainty for all the many other WordPress hosting companies that have "WP" in their names. On a related note, WP Engine is selling WordPress hosting and not WordPress itself as well as claims to have emails from Automattic staff saying using the "WP" short form is okay.

Another point of note here is whether 'WP' is covered by trademark. At some point in the recent past, the wordpressfoundation.org website updated its 'Trademark Policy' page to include an answer as well as some gratuitous remarks:

The abbreviation “WP” is not covered by the WordPress trademarks, but please don’t use it in a way that confuses people. For example, many people think WP Engine is “WordPress Engine” and officially associated with WordPress, which it’s not. They have never once even donated to the WordPress Foundation, despite making billions of revenue on top of WordPress.

Third, Mullenweg's demand that WP Engine cough up 8% of their revenue amounts to a demand for $40 million (Rs 334.5 crore). Considering Automattic has now pinned this demand to the wobbly allegation of wrongful trademark use, WP Engine seems increasingly in the right to dispute and not entertain his demands. Moreover, WP Engine's lawyers' letter suggests Mullenweg gave WP Engine a very small window within which to comply with this demand and, for added measure, allegedly threw in a threat. But then at 10.38 pm on September 24, Mullenweg said this on Reddit:

I would have happily negotiated from there, but they refused to even take a call. Their entire strategy has been to obscure and delay, which they tried to do on Friday. "Can we get the right folks together early next week?" They've been stringing us along for years, I'm the dummy for believing that they actually wanted to do anything. But making it right, now.

The reply to this comment by Reddit user u/ChallengeEuphoric237 is perfect:

If they don't believe they needed to pay, then why would they?

1/ If the fees were for the trademark, why weren't they going to the WordPress Foundation instead of Automattic?

2/ Why do they need to license the WordPress trademark? Stating they allow hosting as a product isn't a violation of trademark law, neither is using WP. You guys used to be an investor in the company for crying out loud.

WordPress and Automattic seem incredibly petty in all of this. Why did you *need* to do it during the keynote? Why did you *need* to make a huge stink at the booth? If this was a legal issue, let the lawyers sort it out instead of dragging the community through the mud. Everyone expected much more from you. I don't use WP Engine's products, but if someone came to me trying to extort 8% of my revenue on some flimsy trademark issue, I wouldn't be very responsive either.

"Can we get the right folks together early next week?"

Did you honestly expect them to agree to a nearly 40 million dollar annual charge via text message when you literally gave them what seems like an hour notice right before your keynote? Would you agree to that? I'm no lawyer, but that whole exchange seems like an exercise in extortion - threatening to destroy someones reputation unless they agree to something monetarily, which is a felony.

Let's see what the courts say, but you've lost a ton of clout in the community over this.

The subtext of Mullenweg's September 21 post seemed to be that private equity is cutting costs in a way that's eating into the aspirations and dues of open source software development. Then again, as many observers in the sector have said, this couldn't be the real issue because private equity is almost everywhere in the WordPress hosting space and singling out WP Engine made little sense. So the sub-subtext seemed to be that Mullenweg was unhappy about WP Engine eating into the revenue streams of WordPress.com and WordPress VIP (Automattic's elite hosting service). But after the events of the three days that followed, that sub-subtext seems likelier to be the whole issue.

On a final note, many people are kicking back with 🍿 and speculating about how this dispute could escalate further. But it's difficult for me personally to be entertained by this. While Mullenweg's September 21 post didn't in hindsight do a good job of communicating what his real argument was, he did suggest there was a problem with a model in which for-profit entities could springboard off the efforts of open-source communities that have volunteered their time and skills without the entities giving back. But dovetailing to u/ChallengeEuphoria237's concluding remark, conversations about that issue vis-à-vis WP Engine are now more unlikely to happen than they were before Mullenweg launched into this "making it right" campaign.

Automattic CEO and WordPress co-developer Matt Mullenweg published a post on September 21 calling WP Engine a “cancer to WordPress”. For the uninitiated: WP Engine is an independent company that provides managed hosting for WordPress sites; WordPress.com is owned by Automattic and it leads the development of WordPress.org. WP Engine’s hosting plans start at $30 a month and it enjoys a good public reputation. Mullenweg’s post however zeroed in on WP Engine’s decision to not record the revisions you’ve made to your posts in your site’s database. This is a basic feature in the WordPress content management system, and based on its absence Mullenweg says:

What WP Engine gives you is not WordPress, it’s something that they’ve chopped up, hacked, butchered to look like WordPress, but actually they’re giving you a cheap knock-off and charging you more for it.

The first thing that struck me about this post was its unusual vehemence, which Mullenweg has typically reserved in the past for more ‘extractive’ platforms like Wix whose actions have also been more readily disagreeable. WP Engine has disabled revisions but as Mullenweg himself pointed out it doesn’t hide this fact. It’s available to view on the ‘Platform Settings’ support page. Equally, WP Engine also offers daily backups; you can readily restore one of them and go back to a previous ‘state’.

Second, Mullenweg accuses WP Engine of “butchering” WordPress but this is stretching it. I understand where he’s coming from, of course: WP Engine is advertising WordPress hosting but it doesn’t come with one of the CMS’s basic features, and which WP Engine doesn’t hide but doesn’t really advertise either. This isn't just really far removed from “butchering” (much less in public), it's also dishonest: WP Engine didn't modify WordPress's core, it simply turned off a setting that was available to turn off.

WP Engine’s stated reason is that post revisions increase database costs that the company would like to keep down. Mullenweg interprets this to mean WP Engine wants “to avoid paying to store that data”. Well, yeah, and that’s okay, right? I can’t claim to be aware of all the trade-offs that determined WP Engine’s price points but turning off a feature to keep costs down and reactivating it upon request for individual users seems fair.

In fact, what really gets my goat is Mullenweg’s language, especially around how much WP Engine charges. He writes:

They are strip-mining the WordPress ecosystem, giving our users a crappier experience so they can make more money.

WordPress.com offers a very similar deal to its customers. (WordPress.com is Automattic’s platform for users where they can pay the company to host WordPress sites for them.) In the US, you’ll need to pay at least $25 a month (billed yearly) to be able to upload custom themes and plugins to your site. All the plans below that rate don’t have this option. You also need this plan to access and jump back to different points of your site’s revision history.

Does this mean WordPress.com is “strip-mining” its users to avoid paying for the infrastructure required for those features? Or is it offering fewer features at lower price points because that’s how it can make its business work? I used to be happy that WordPress.com offers a $48 a year plan with fewer features because I didn’t need them — just as well as WP Engine seems to have determined it can charge its customers less by disabling revision history by default.

(I’m not so happy now because WordPress.com moved detailed site analytics — anything more than hits to posts — from the free plan to the Premium plan, which costs $96 a year.)

It also comes across as disingenuous for Mullenweg to say the “cancer” a la WP Engine will spread if left unchecked. He himself writes no WordPress host listed on WordPress.org’s recommended hosts page has disabled revisions history — but is he aware of the public reputation of these hosts, their predatory pricing habits, and their lousy customer service? Please take a look at Kevin Ohashi’s Review Signal website or r/webhosting. Cheap WordPress in return for a crappy hosting experience is the cancer that’s already spread because WordPress didn’t address it.

(It’s the reason I switched to composing my posts offline on MarsEdit, banking on its backup features, and giving up on my expectations of hosts including WordPress.com.)

It’s unfair to accuse companies of “strip-mining” WordPress so hosting providers can avail users a spam-free, crap-free hosting experience that’s also affordable. In fact, given how flimsy many of Mullenweg’s arguments seem to be, they’re probably directed at some other deeper issue — perhaps what he perceives to be WP Engine not contributing enough back to the open source ecosystem?