We have an extended power outage coming up, one that's long enough that I think we want a backup MX that can stay up during it. I've been building out a stand-alone duplicate of our current inbound mail gateway, and today I added a lower priority DNS MX record that points to it. What happened next is predictable:

This is my absolutely not surprised face that mere moments after I add a secondary MX to one of our zones, various IPs show up to poke its SMTP port, despite our primary MX being up (and the backup MX not actually running a SMTP server right now).

Admittedly, there's a reason for some use of our backup MX, which I discovered after I started the MTA on the backup MX machine:

Oops, I have to retract some of my 'the spammers are showing up on schedule' snark, because our primary MX greylists people sometime and if the primary MX is 4xx'ing things, trying the backup MX is reasonable.

(But surprise, the backup MX will greylist you too because our MXes are running the same configuration.)

Another case where things appear to have more or less legitimately shifted over to the backup MX was when one particular amazonses.com IP address opened up so many simultaneous connections to our primary MX that our primary MX started giving that IP temporary failures on connection. Trying the backup MX when the primary MX gives you an immediate 4xx is reasonable.

As far as I can tell, this isn't general SMTP probes against DNS names or IP addresses:

I did some more digging using our firewall PF logs and it appears pretty definite that some people showed up to do SMTP authentication probes only after this host appeared in DNS MX and got a TLS certificate. It's possible that the TLS certificate is the trigger for SMTP auth attempts, but they're very bad SMTP auth attempts (they aren't starting TLS, for a start, and this backup MX doesn't do SMTP auth).

Most of the sending machines that showed up were clearly bad, and many of them were rejected (or at the least sent things that got extremely high spam scores). Very few of them showed any signs of having tried to contact our primary MX. All of this matches what I think of as the expected behavior, where spammers and other bad actors hope that your backup MX is less well protected than your primary MX, so they prefer to talk to it if they can in the hopes that they'll get more bad stuff through.

(I've heard this story about backup MXes for a long time, but I never had a backup MX around to see this happen. It's nice, in a way, to have this story confirmed right in front of me.)

Some sending machines are more mysterious. For example, one outlook.com machine contacted the backup MX instead of the primary MX for no clear reason that I can see. These days, it's entirely possible that there was a transient network glitch on the path between that machine and us when it was trying to contact the primary MX, so it tried the secondary after its first connection glitched out.

Given all of this, if I was building a backup MX for full time use, it would be tempting to build a system where the MTA (mail server) was only enabled once the backup MX detected that the primary MX wasn't responding. Depending on taste, I could make the backup MX's MTA generate 4xx errors on connection or simply have it not be running at all so people got 'connection refused' if they tried. Checking once a minute or once every few minutes would be fine for our intended uses.

(In our planned one time use, we'll just enable and disable the backup MX's MTA by hand.)

To simplify, DMARC is based on the domain in the 'From:' header, and what policy (if any) that domain specifies. As I've written about (and rediscovered) more than once (here and here), DMARC will look up the DNS record for the DMARC policy in exactly one of two places, either in the exact From: domain or on the organization's top level domain. In other words, if a message has a From: of 'someone@breaking.news.example.org', a receiver will first look for a DMARC TXT DNS record with the name _dmarc.breaking.news.example.org and then one with the name _dmarc.example.org.

(But there will be no lookup for _dmarc.news.example.org.)

DMARC also has the concept of policy inheritance, where the example.org DMARC DNS TXT record can specify a different DMARC policy for the organizational domain than for subdomains that don't have their own policy. For example, example.org could specify 'p=reject; sp=none' to say that 'From: user@example.org' should be rejected if it fails DMARC but it has no views on a default for 'From: user@news.example.org'.

If you're an innocent person, you might think that if your organization has 'sp=none' on its organization policy, you don't have to be concerned about the DMARC (and DKIM, and SPF) behavior of sub-names that don't have their own DMARC records, including hosts that send as 'From: local-account@host.dept.example.org'. Your organizational policy says 'sp=none', meaning don't do anything with sub-names for DMARC, and surely everyone will follow that.

This is unfortunately not quite true in an environment where people care about DKIM results regardless of DMARC policy settings. The problem is DKIM (and SPF) alignment. Under relaxed DKIM alignment, a 'From: flash@eng.news.example.org' would pass if it's DKIM signed by anything in example,org, for example 'eng.example.org'. Under strict DKIM alignment, it must be signed specifically by 'eng.news.example.org'.

The choice of what DKIM alignment to require is not a 'policy' and is not covered by 'p=' or 'sp=' in DMARC DNS TXT records. It's instead covered by a separate parameter, 'adkim=', and there is no 'sadkim=' parameter that only applies to subdomains. This means that there's no way for example.org to change the alignment policy for just 'From: user@example.org'; the moment they set 'adkim=s' in the _dmarc.example.org DNS TXT record, all sub-names without their own _dmarc.<whatever> records also switch to strict DKIM alignment. Even if the top level domain specifies 'sp=none', various mail systems out there may actively reject your mail because they no longer consider it properly aligned or increase their suspicion score a bit due to the lack of alignment (in some views your mail went from 'properly DKIM signed' to 'not properly DKIM signed').

The only way to deal with this is the same as with policy inheritance. Any host or domain name within your (sub-)organization that appears in From: headers must have its own valid DMARC DNS TXT record. If you want strict DKIM alignment you need to set that as 'adkim=s'. If you want relaxed alignment in theory that's the default but you might find it clearer to explicitly set 'adkim=r' (and probably 'aspf=r', also for clarity).

(Setting alignment explicitly makes it clear to other people and future you that you're deliberately choosing an alignment that might wind up different from your top level organizational alignment.)

PS: As far as I can see this is the behavior the DMARC RFC implicitly requires for all DMARC settings other than 'p=' (which has the 'sp=' version), but I could be wrong and missing something.

DMARC has a feature where you can request that other mail systems send you aggregate reports about the DMARC results that they observed for email claiming to be from you. If you're a large institution with a sprawling, complex, multi-party mail environment and you're considering trying to make your DMARC policy stricter, it's very useful to get as many DMARC reports from as many people as possible. Especially, 'you' (in a broad sense) probably want to get as much information from mail systems run by sub-units as possible, and if you're a sub-unit, you want to report DMARC information up to the organization so they have as much visibility into what's going on as possible.

In related news, I've been looking into making our mail system send out DMARC reports, and I had what was in retrospect a predictable learning experience:

Today's discovery: if you want to helpfully send out DMARC reports to people who ask for them and you operate even a moderate sized email system, you're going to need to use a dedicated sending server and you probably don't want to. Because a) you'll be sending a lot of email messages and b) a lot of them will bounce because people's DMARC records are inaccurate and c) a decent number of them will camp out in your mail queue because see b, they're trying to go to non-responsive hosts.

Really, all of this DMARC reporting nonsense was predictable from first (Internet) principles, but I didn't think about it and was just optimistic when I turned our reporting on for local reasons. Of course people are going to screw up their DMARC reporting information (or for spammers, just make it up), they screw everything up and DMARC data will be no exception.

(Or they take systems and email addresses out of service without updating their DMARC records.)

If you operate even a somewhat modest email system that gets a wide variety of email, as we do, it doesn't take very long to receive email from hundreds of From: domains that have DMARC records in DNS that request reports. When you generate your DMARC reports (whether once a day or more often), you'll send out hundreds of email messages to those report addresses. If you send them through your regular outgoing email system, you'll have a sudden influx of a lot of messages and you may trigger any anti-flood ratelimits you have. Once your reporting system has upended those hundreds of reports into your mail system, your mail system has to process through them; some of them will be delivered promptly, some of them will bounce (either directly or inside the remote mail system you hand them off to), and some of them will be theoretically destined for (currently) non-responsive hosts and thus will clog up your mail queue with repeated delivery attempts. If you're sending these reports through a general purpose mail system, your mail queue probably has a long timeout for stalled email, which is not really what you want in this case; your DMARC reports are more like 'best effort one time delivery attempt and then throw the message away' email. If this report doesn't get through and the issue is transient, you'll keep getting email with that From: domain and eventually one of your reports will go through. DMARC reports are definitely not 'gotta deliver them all' email.

So in my view, you're almost certainly going to have to be selective about what domains you send DMARC reports for. If you're considering this and you can, it may help to trawl your logs to see what domains are failing DMARC checks and pick out the ones you care about (such as, say, your organization's overall domain or domains). It's somewhat useful to report even successful DMARC results (where the email passes DMARC checks), but if you're considering acting on DMARC results, it's important to get false negatives fixed. If you want to send DMARC reports to everyone, you'll want to set up a custom mail system, perhaps on the DMARC local machine, which blasts everything out, efficiently handles potentially large queues and fast submission rates, and discards queued messages quickly (and obviously doesn't send you any bounces).

(Sending through a completely separate mail system also avoids the possibility that someone will decide to put your regular system on a blocklist because of your high rate of DMARC report email.)

PS: Some of those hundreds of From: domains with DMARC records that request reports will be spammer domains; I assume that putting a 'rua=' into your DMARC record makes it look more legitimate to (some) receiving systems. Spammers sending from their own domains can DKIM sign their messages, but having working reporting addresses requires extra work and extra exposure. And of course spammers often rotate through domains rapidly.

When you use DMARC, you get to specify a policy that people should apply to email that claims to be from your domain but doesn't pass DMARC checks (people are under no obligation to pay attention to this and they may opt to be stricter). These policies are set in DNS TXT records, and in casual use we can say that the policies of subdomains in your domain can be 'inherited'. This recently confused me and now I have some answers.

Your top level domain can specify a separate policy for itself (eg 'user@example.org') and subdomains (eg 'user@foo.example.org'); these are the 'p=' and 'sp=' bits in a DMARC DNS TXT record. Your domain's subdomain policy is used only for subdomains that don't set a policy themselves; an explicitly set subdomain policy overrides the domain policy, for better or worse. If your organization wants to force some minimum DMARC policy, you can't do it with a simple DNS record; you have to somehow forbid subdomains from publishing their own conflicting DMARC policies in your DNS.

The flipside of this is that it's not as bad as it could be to set a strict subdomain policy in your domain DMARC record, because subdomains that care can override it (and may already be doing so implicitly if they've published DMARC records themselves).

However, strictly speaking DMARC policies aren't inherited as we usually think about it. Instead, as I once knew but forgot since then, people using DMARC will check for an applicable policy in only two places: on the direct domain or host name that they care about, and on your organization's top level domain. What this means in concrete terms is that if example.org and foo.example.org both have DMARC records and someone sends email as 'user@bar.foo.example.org', the foo.example.org DMARC record won't be checked. Instead, people will look for DMARC only at 'bar.foo.example.org' (where any regular 'p=' policy will be used) and at 'example.org' (where the subdomain policy, 'sp=', will be used).

(As a corollary, a 'sp=' policy setting in the foo.example.org DMARC record will never be used.)

One place this gets especially interesting is if people send email using the domain 'nonexistent.foo.example.org' in the From: header (either from inside or outside your organization). Since this host name isn't in DNS, it has no DMARC policy of its own, and so people will go straight to the 'example.org' subdomain policy without even looking at the policy of 'foo.example.org'.

(Since traditional DNS wildcard records can only wildcard the leftmost label and DMARC records are looked up on a special '_dmarc.' DNS sub-name, it's not simple to give arbitrary names under your subdomain a DMARC policy.)

I'm a long time advocate of using expendable email addresses any time you have to give someone an email address (and then making sure you can turn them off or more broadly apply filters to them). However, some of the time I've trusted the people who were asking for an email address, didn't have an expendable address already prepared for them, and gave them my regular email address. Today I discovered (or realized) another reason to not do this and to use expendable addresses for absolutely everything, and it's not the usual reason of "the people you gave your email address to might get compromised and have their address collection extracted and sold to spammers". The new problem is mailing service providers, such as Mailchimp.

It's guaranteed that some amount of spammers make use of big mailing service providers, so you will periodically get spam email to any exposed email address, most likely including your real, primary one from such MSPs. At the same time, these days it's quite likely that anyone you give your email address to will at some point wind up using an MSP, if only to send out a cheerful notification of, say, "we moved from street address A to street address B, please remember for planning your next appointment" (because if you want to send out such a mass mailing, you basically have to outsource it to an MSP to get it done, even if you normally use, eg, GMail for your regular organizational activities).

If you've given innocent trustworthy organizations your main email address, it's potentially dangerous or impossible to block a particular MSP from sending email to it. In searching your email archive, you may find that such an organization is already using the MSP to send you stuff that you want, or for big MSPs you might decide that the odds are too bad. But if you've given separate expendable email addresses to all such organizations, you know that they're not going to be sending anything to your main email address, including through some MSP that you've just got spam from, and it's much safer to block that MSP's access to your main email address.

This issue hadn't occurred to me back when I apparently gave one organization my main email address, but it became relevant recently. So now I'm writing it down, if only for my future self as a reminder of why I don't want to do that.

Today I had reason to report spam to Mailchimp (some of you are laughing already, I know). As I usually do, I forwarded the spam message we'd received to them as a message/rfc822 MIME part, with a prequel plain text part saying that it was spam. Forwarding email as a MIME message/rfc822 part is unambiguously the correct way to do so. It's in the MIME RFCs, if done properly (by the client) it automatically includes all headers, and because it's a proper MIME part, tools can recognize the forwarded email message, scan over just it, and so on.

So of course Mailchimp sent me back an autoreply to the effect that they couldn't find any spam mail message in my report. They're not the only people who've replied this way, although sometimes the reply says "we couldn't handle this .eml attachment". So I had to re-forward the spam message in what I called literal plaintext format. This time around either some human or some piece of software found the information and maybe correctly interpreted it.

I think it's perfectly fine and maybe even praiseworthy when email abuse handling systems (and people) are willing to accept these literal plaintext format forwarded spam messages. The more formats you accept abuse reports in, the better. But every abuse handling system should accept MIME message/rfc822 format messages too, as a minimum thing. Not just because it's a standard, but also because it's what a certain amount of mail clients will produce by default if you ask them to forward a message. If you refuse to accept these messages, you're reducing the amount of abuse reports you'll accept, for arbitrary (but of course ostensibly convenient for you) reasons.

I know, I'm tilting at windmills. Mailchimp and all of the other big places doing this don't care one bit what I want and may or may not even do anything when I send them reports.

(I suspect that many places see reducing the number of 'valid' abuse reports they receive as a good thing, so the more hoops they can get away with and the more reports they can reject, the better. In theory this is self-defeating in the long run, but in practice that hasn't worked with the big offenders so far.)

TLS SNI (Server Name Indication) is something that a significant minority of sending mail servers use when they do TLS with SMTP. One of the reasons that it's not used more generally is apparently that there's confusion about what TLS SNI name to use. Based on our logs, in practice essentially everyone using TLS SNI uses the MX target name as the SNI name; if something is MX'd to 'inbound.example.org', then sending mailers with send the SNI name 'inbound.example.org'.

One other option for the TLS SNI name is the domain name of the recipient. Using the TLS SNI of the recipient domain would let SMTP frontends route the connection to an appropriate backend in still encrypted form, although you'd have to use a custom protocol. If you then required a matching name in the server TLS certificate, you'd also have assurance that you were delivering the email to a mail server that should handle that domain's mail, rather than having someone intercept your DNS MX query and provide their own server as the place to send 'example.org' mail. However, this has some problems.

First, it means that a sending mailer couldn't aggregate email messages to multiple target domains all hosted by the same MX target into a single connection. I suspect that this isn't a big issue and most email isn't aggregated this way in the first place. More importantly, if the receiving server's TLS certificate had to match the SNI it received, you would need to equip your inbound mail servers with a collection of potentially high value TLS certificates for your bare domain names. The 'inbound.example.org' server would need a TLS server certificate for 'example.org' (and maybe 'example.net' if you had both and handled both in the same inbound server). In the current state of TLS, I don't believe this TLS certificate could be scoped down so that it couldn't be used for HTTPS.

This would also be troublesome for outsourcing your email handling to someone. If you outsourced your email to example.org, you would have to give them a TLS certificate (or a series of TLS certificates) for your bare domain, so they could handle email for it with the right TLS certificate.

Sending the target MX name as the TLS SNI name is less useful for some things and is more prone to DNS interception and other ways to tamper with DNS results (assuming there's no DNSSEC in action). But it has the virtue that it's simple to implement on both sides and you don't have to give your inbound mail server access to potentially sensitive TLS certificates. It can have a TLS certificate just for its own name, where the only thing this can really be used for is mail just because that's the only thing the server does. And of course it works well with outsourced mail handling, because the MX target server only needs a TLS certificate for its own name, which its organization should be able to easily provide.

Arguably part of the confusion here is an uncertainly over what using TLS SNI during SMTP STARTTLS is supposed to achieve and what security properties we want from the resulting TLS connection. In HTTPS TLS, using SNI has a clear purpose; it lets you handle multiple websites, each with their own TLS certificate, on a single IP, and you want the TLS certificate to be valid for the TLS SNI name. I don't think we have any such clarity for TLS SNI and general server TLS certificate validation in SMTP. For example, are we using TLS certificates to prove that the SMTP server you're talking to is who you think it is (the MX target name), or that it's authorized to handle email for a particular domain (the domain name of the destination)?

(This elaborates on a Fediverse post in a discussion of this area.)

This is a companion to my entry on the types of TLS seen for incoming email; this is for May 2025 because that's when the data I'm using comes from. This data covers nine days and about 12,800 external mail deliveries that originated from people (instead of from things like mail forwarding), and I'm going to be rounding numbers off for my own reasons.

Of those external deliveries, almost all of them used some form of TLS; basically 99% (call it 12,675, although it turns out a bunch of those should be ignored for reasons beyond the scope of this entry). Almost all of the 'real' messages used TLS 1.3; 89% used TLS 1.3 and 11% used TLS 1.2, with no other TLS versions used. Interestingly, the outgoing top level signature schemes are different than the incoming ones:

  5240  X=TLS1.3:ECDHE_SECP256R1
  2530  X=TLS1.3:ECDHE_X25519
   660  X=TLS1.2:ECDHE_SECP256R1
   220  X=TLS1.2:ECDHE_X25519
    18  X=TLS1.2:RSA
     5  X=TLS1.2:ECDHE_SECP384R1
     4  X=TLS1.2:ECDHE_SECP521R1
     2  X=TLS1.3:ECDHE_SECP384R1
     2  X=TLS1.2:DHE_CUSTOM2048

The destinations that used TLS 1.2 are much more assorted than I would have expected and they make me wonder about the cipher preferences that our Ubuntu 22.04 version of Exim is telling servers about. On the other hand, some of the surprising ones are symmetrical; for example, people at Amazon appear to genuinely be using TLS 1.2 both when receiving email from us and when sending email to their correspondents here (amazonses.com uses TLS 1.3 for outgoing email, but amazon.com doesn't seem to).

We send a lot of email to some places. One of them is hosted by Microsoft, and uses TLS 1.3 with ECDHE SECP256R1; another is GMail, which (with us) uses TLS 1.3 with ECDHE X25519. These distort the overall outgoing statistics, although there is probably a similar effect for our incoming email. Now that I'm poking at these logs, I've wound up feeling that a real analysis would have to look at the organizations running the MX targets of the domains (which is too much work for me for now).

The non-TLS destinations are mostly some groups within the university who seem to still have (old) mail servers that haven't been set up with TLS for various reasons. There are a few outside organizations, including a university or two that surprise me.

(These statistics are less interesting than I was hoping, but so it goes sometimes. I don't know unless I go look.)

Back in April 2023 I did some statistics on what versions of TLS our external SMTP email gateway was seeing. Today, for reasons outside of the scope of this entry, I feel like revisiting those numbers to show how things have changed (somewhat). As with the first set of numbers, these cover the previous nine days of data to us, a fairly large computer science department in a fairly large university.

(Conveniently, our external SMTP gateway is still running Ubuntu 22.04, as it was two years ago, and so it still has the same selection of TLS versions and cipher suites and so on.)

Over the past nine days, we've received 90,938 email messages, of which 74,256 used some version of TLS, so roughly 82% of our incoming email is encrypted and 18% isn't. This latter number (and percent) is not really great, but it is what it is and it's substantially smaller than it was two years ago. What Exim reports for TLS versions breaks down as follows:

 58596  X=TLS1.3   [79%]
 15509  X=TLS1.2   [21%]
   135  X=TLS1.0
    16  X=TLS1.1

The TLS 1.0 email appears to be all or almost all from spammers, and I think it's mostly from on particularly prolific spam source (that revolves IPs, sending domains, and so on). The TLS 1.1 email is from a small handful of places, and anyway there's almost none of it. The TLS 1.2 email is a substantial portion and appears to come from a number of places, including some hosting providers, various significant universities, ieee.org, lists.ubuntu.com, and others. If we ignore various machines within the university, the non-TLS email appears to be mostly but not entirely spammers; one notable real non-TLS source is Air Canada.

The heartening thing to me is that in two years, incoming TLS versions have switched so that TLS 1.3 is dominant and TLS 1.2 has shrunk to only a fifth. I'm not sure I would have guessed that things would change that fast.

Exim conveniently formats its TLS information so I can show a top level view of the broad signature schemes in use:

 48011  X=TLS1.3:ECDHE_X25519
 11774  X=TLS1.2:ECDHE_SECP256R1
  9968  X=TLS1.3:ECDHE_SECP384R1
  1799  X=TLS1.2:ECDHE_X25519
  1631  X=TLS1.2:ECDHE_SECP521R1
   617  X=TLS1.3:ECDHE_SECP256R1
   159  X=TLS1.2:ECDHE_SECP384R1
   146  X=TLS1.2:RSA
   129  X=TLS1.0:ECDHE_SECP256R1
    12  X=TLS1.1:ECDHE_SECP521R1
     4  X=TLS1.1:RSA
     4  X=TLS1.0:RSA
     2  X=TLS1.0:ECDHE_SECP521R1

Pretty clearly, in TLS 1.3 ECDHE with X25519 has decisively won (at least in inbound SMTP) over the NIST curves, although there are still a decent number of holdouts. This dominance isn't there in TLS 1.2, where instead ECDHE with X25519 is a minority position and the dominant one is SECP256R1.

Overall there were 31 different full cipher suites used, and so I'll give a little (partial) breakdown by protocol:

 24653  X=TLS1.3: ECDHE_X25519 RSA_PSS_RSAE_SHA256 AES_256_GCM: 256
 23083  X=TLS1.3: ECDHE_X25519 RSA_PSS_RSAE_SHA256 AES_128_GCM: 128
  9968  X=TLS1.3: ECDHE_SECP384R1 RSA_PSS_RSAE_SHA256 AES_256_GCM: 256
   617  X=TLS1.3: ECDHE_SECP256R1 RSA_PSS_RSAE_SHA256 AES_256_GCM: 256
   275  X=TLS1.3: ECDHE_X25519 RSA_PSS_RSAE_SHA512 AES_256_GCM: 256

  8199  X=TLS1.2: ECDHE_SECP256R1 RSA_SHA512 AES_256_GCM: 256
  1978  X=TLS1.2: ECDHE_SECP256R1 RSA_SHA256 AES_128_GCM: 128
  1590  X=TLS1.2: ECDHE_SECP521R1 RSA_SHA512 AES_256_GCM: 256
  1428  X=TLS1.2: ECDHE_SECP256R1 RSA_SHA512 AES_128_GCM: 128
   738  X=TLS1.2: ECDHE_X25519 RSA_PSS_RSAE_SHA256 AES_128_GCM: 128
 [... 21 different ones in total ...]
     1  X=TLS1.2: RSA AES_128_CBC SHA1: 128

    12  X=TLS1.1: ECDHE_SECP521R1 RSA_SHA1 AES_256_CBC SHA1: 256
     4  X=TLS1.1: RSA AES_256_CBC SHA1: 256

   129  X=TLS1.0: ECDHE_SECP256R1 RSA_SHA1 AES_256_CBC SHA1: 256
     4  X=TLS1.0: RSA AES_256_CBC SHA1: 256
     2  X=TLS1.0: ECDHE_SECP521R1 RSA_SHA1 AES_256_CBC SHA1: 256

(This has been slightly reformatted from how Exim presents its ciphers to create more word breaks.)

The least used cipher is the TLS 1.2 one above that was used once. It's somewhat amusing to me that TLS 1.3 has such an even division between X25519 ECDHE with 128-bit or 256-bit AES GCM (which was also there two years ago).

Sidebar: the TLS 1.2 RSA ciphers

Here they are:

    97  X=TLS1.2:RSA__AES_256_CBC__SHA1:256
    48  X=TLS1.2:RSA__AES_256_GCM:256
     1  X=TLS1.2:RSA__AES_128_CBC__SHA1:128

As before, I don't know how horrified I should be.

Over on the Fediverse, I shared a discovery:

This is my face when rspamd will apparently pattern-match a mention of 'test@test' in the body of an email, extract 'test', try that against the multi.surbl.org DNS blocklist (which includes it), and decide that incoming email is spam as a result.

Although I didn't mention it in the post, I assume that rspamd's goal is to extract the domain from email addresses and see if the domain is 'bad'. This handles a not uncommon pattern of spammer behavior where they send email from a throwaway setup but direct your further email to their long term address. One sees similar things with URLs, and I believe that rspamd will extract domains from URLs in messages as well.

(Rspamd is what we currently use for scoring email for spam, for various reasons beyond the scope of this entry.)

The sign of this problem happening was message summary lines in the rspamd log that included annotations like (with a line split and spacing for clarity):

[...] MW_SURBL_MULTI(7.50){test:email;},
PH_SURBL_MULTI(5.00){test:email;} [...]

As I understand it, the 'test:email' bit means that the thing being looked up in multi.surbl.org was 'test' and it came from the email message (I don't know if it's specifically the body of the email message or this could also have been in the headers). The SURBL reasonably lists 'test' for, presumably, testing purposes, much like many IP based DNSBLs list various 127.0.0.* IPs. Extracting a dot-less 'domain' from a plain text email message is a bit aggressive, but we get the rspamd that we get.

(You might wonder where 'test@test' comes from; the answer is that in Toronto it's a special DSL realm that's potentially useful for troubleshooting your DSL (also).)

Fortunately rspamd allows exceptions. If your rspamd configuration directory is /etc/rspamd as normal, you can put a 'map' file of SURBL exceptions at /etc/rspamd/local.d/map.d/surbl-whitelist.inc.local. You can discover this location by reading modules.d/rbl.conf, which you can find by grep'ing the entire /etc/rspamd tree for 'surbl' (yes, sometimes I use brute force). The best documentation on what you put into maps that I could find is "Maps content" in the multimap module documentation; the simple version is that you appear to put one domain per line and comment lines are allowed, starting with '#'.

(As far as I could tell from our experience, rspamd noticed the existence of our new surbl-whitelist.inc.local file all on its own, with no restart or reload necessary.)

Yesterday I wrote about how we got hit by an alarmingly well-prepared phish spammer. This spammer sent a moderate amount of spam through us, in two batches; most of it was immediately delivered or bounced (and was effectively lost), but we managed to capture one message due to delivery problems. We can't be definite from a single captured spam message (and our logs suggesting that the other messages were similar to it), but it's at least suggestive.

The single captured email message has two PDFs and a text portion; as far as I can tell the PDFs are harmless (apart from their text contents), with no links or other embedded things. The text portion claims to be a series of (top replying) email messages about the nominal sender of the message getting an invoice paid, and the PDFs are an invoice for vague professional services for $49,700 (US dollars, implicitly), with a bank's name, a bank routing number and an account number, and a US IRS W-9 form for the person supposedly asking for their invoice to be paid, complete with an address and a US Social Security number. The PDF requests that you 'send a copy of the remittance to <email address>', where the domain has no website and its mail is hosted by Google. Based on some Internet searches, the PDF's bank routing number is correct for the bank, although of course who knows who the account number goes to.

The very obvious thing to say is that if even a single recipient out of the just under three hundred this spam was sent to follows the directions and sends an invoice payment, this will have been a decently lucrative phish spam (assuming that all of the spam messages were pushing the same scam, and the spammer can extract the money). If several of them did, this could be extremely lucrative, more than lucrative enough to justify dozens or hundreds of hours of research on both the ultimate targets (to determine who at various domains to send email to, what names of bosses to put in the email, and so on) and access methods (ie, how to use our VPNs).

Further, it seems possible that the person whose name was on the invoice, the email, and the W-9 is real and had their identity stolen, complete with their current address and US social security number. If this is the case, the person may receive an unpleasant surprise the next time they have to interact with the US IRS, since the IRS may well have data from companies claiming that this person was paid income that, well, they weren't. I can imagine a more advanced version of the scam where the spammer actually opened an account in this person's name at the bank in the invoice, and is now routing their fraudulently obtained invoice payments through it.

(There are likely all sorts of other possibilities for how the spammer might be extracting invoice payment money, and all of this assumes that the PDFs themselves don't contain undetected malware that is simply inactive in my Linux command line based PDF viewing environment.)

Yesterday evening, we were hit by a run of phish spam that I would call 'vaguely customized' for us, for example the display name in the From: header was "U of T | CS Dept" (but then the actual email address was that of the compromised account elsewhere that was used to send the phish spam). The destination addresses here weren't particularly well chosen, and some of them didn't even exist. So far, so normal. One person here fell for the phish spam that evening but realized it almost immediately and promptly changed their password. Today that person got in touch with us because they'd started receiving email bounces for (spam) email that they hadn't sent. Investigation showed that the messages were being sent through us, but in an alarmingly clever way.

We have a local VPN service for people, and this VPN service requires a different password from your regular (Unix and IMAP and etc) password. People connecting through our VPN have access to an internal-only SMTP gateway machine that doesn't require SMTP authentication. As far as we can tell, in the quite short interval between when the person fell for the phish and then changed their password, the phish spam attacker used the main password they'd just stolen to register the person for our VPN and obtain a VPN password (which we don't reset on Unix password changes). They then connected to the VPN using their stolen credentials and used the VPN to send spam email through our internal-only SMTP gateway (initially last evening and then again today, at which point they were detected).

Based on some log evidence, I think that the phish spammer first tried to use authenticated SMTP but failed due to the password change, then fell back on the VPN access. Even if VPN access hadn't been their primary plan, they worked very fast to secure themselves an additional access method. It seems extremely likely that the attacker had already researched our mail and VPN environment before they sent their initial phish spam, since they knew exactly where to go and what to do.

If phish spammers are increasingly going to be this well prepared and clever, we're going to have to be prepared for that on our side. Until now, we hadn't really thought about the possibility of phish spammers gaining VPN access; previous phish spammers have exploited some combination of webmail and authenticated SMTP.

(We're also going to need to be more concerned about other methods of obtaining persistent account access, such as adding new SSH authorized keys to the Unix login. This attacker didn't attempt any sort of SSH access.)

Once upon a time (a long time ago), filtering and rejecting email based on the SMTP envelope sender (the SMTP MAIL FROM) was a generally sufficient mechanism to deal with many repeat spam sources. It didn't deal with all of them but many used their own domain in the envelope sender, even if they send from a variety of different IP addresses. Unfortunately, the rise of (certain) mail service providers has increasingly limited the usefulness of envelope sender address filtering, because an increasing number of the big providers use their own domains for the envelope sender addresses of all outgoing email. Unless you feel like blocking the provider entirely (often this isn't feasible, even on an individual basis), rejecting based on the envelope sender doesn't do you any good here.

This has made it increasingly useful to be able to do SMTP time rejection (and general filtering) based on the 'From:' header address. Many mail sending services will put the real spam source's email address in the From: and at least the top level domain of this will be consistent for a particular source, which means that you can use it to reject some of their customers but accept others. These days, MTAs (mail transfer agents) generally give you an opportunity to reject messages at the SMTP DATA phase, after you've received the headers and message body, so you can use this to check the From: header address.

(If you're applying per-destination filtering, you have the SMTP DATA error problem and may only be able to do this filtering if the incoming email has only a single recipient. Conveniently, the mail service providers that commonly obfuscate the envelope sender address usually send messages with only a single recipient for various reasons, including VERP or at least something that looks like it.)

I feel that From: address filtering works best on pseudo-legitimate sources of repeat spam, such as companies that are sending you marketing email without consent. These are the senders that are least likely to vary their top level domain, because they have a business and want to look legitimate, be found at a consistent address, and build up reputation. These are also the sources of unwanted email that are the least likely to be dropped as customers by mail service providers (for a collection of likely reasons that are beyond the scope of this entry).

There are plenty of potential limitations on From: header address filtering. Bad actors can put various sorts of badly formed garbage in the From:, you definitely have to parse it (ideally your MTA will provide this as a built-in), and I believe that it still technically might have multiple addresses. But as a heuristic for rejecting unwanted mail, all of this is not a serious problem. Most From: addresses are well formed and good, especially now that DMARC and DKIM are increasingly required if you want the large providers to accept your email.

(DKIM signing in 'alignment' with the From: header is increasingly mandatory in practice, which requires that the From: header has to be well formed. I don't know how Google and company react to badly formed or peculiar From: headers, but I doubt it helps your email appear in people's inboxes.)

PS: While you can filter or discard email based on the From: header in a variety of places, I like rejecting at SMTP time and it's possible that SMTP rejections at DATA time will trigger anti-spam precautions in the mail service providers (it's a possible signal of badness in the message).

As I mentioned the other day, in late September my home ISP changed my IPv6 allocation from a /64 to a different /56, but kept the old /64 still routing to me. I promptly changed all DNS entries that referred to the old IPv6 address to the new IPv6 address. One of the things that my home machine runs is my 'sinkhole' SMTP server, which has a DNS MX entry pointing to it. This server tracks which local IP address was connected to, and it does periodically receive spam and see probes.

Since this server was most recently restarted on November 10th, it's seen about the same volume of connections to each IPv6 address, the old one (which hasn't been present in DNS for more than a month) and the new one (present in DNS). Some of this activity appears to be from Internet scanning efforts, which I will charitably assume are intending to do good and which have arguable reasons to keep scanning any IPv6 address that they've seen respond. Other connections seem less likely to be innocent.

I'm pretty certain I've seen this behavior for IPv4 addresses long ago (I might even have written it up here, although I can't find an entry right now), so in a sense it doesn't surprise me. Some spammers and other systems apparently do DNS lookups only infrequently and save the IP addresses (both IPv4 and apparently IPv6) that they see, then use them for a long time. Still, it's a more modern world, so I'd sort of hoped that any spammer with software that could deal with IPv6 would handle DNS lookups better.

On the one hand, it's not like holding on to the IP addresses of old mail servers is likely to do spammers much good. If the IP address of a mail server changes, it's very likely that the old IP address will stop working before too long. On the other hand, presumably this mostly doesn't hurt because most mail servers don't change IP addresses very often. Usually the IP address you looked up two months ago (or more) is still good.

Suppose, hypothetically, that you're a clever email spammer and you'd like to increase the legitimacy of your (spam) email by giving it a good DKIM signature, such as a DKIM signature from a reasonably reputable provider of mailing list services. The straightforward way to do this is to sign up to the provider, upload your spam list, and send your email to it; the provider will DKIM sign your message on the way through. However, if you do this you'll generally get your service cancelled and have to go through a bunch of hassles to get your next signup set up. Unfortunately for everyone else, it's possible for spammers to do better.

The spammer starts by signing up to the provider and setting up a mailing list. However, they don't upload a bunch of addresses to it. Instead, they set the list to be as firmly anti-spam, 'confirmed opt in through the provider' as the provider supports. Then they use a bunch of email addresses under their own control to sign up to the mailing list, opt in to everything, and so on. They may then even spend a bit of time sending marketing emails to their captive mailing list of their own addresses, which will of course not complain in the least.

Then the spammer sends their real spam mailing to the mailing list, goes to one of their captive addresses, copies the entire raw message, headers and all, and strips out the 'Received:' headers that come from it leaving the mailing list provider. Then they go to their (rented) spam sending infrastructure and queue up a bunch of spam sending of this message to the real targets, setting it to have a '<>' null SMTP MAIL FROM. This message has a valid DKIM signature put on by the mailing list provider and its SMTP envelope sender is not (quite) in conflict with it. The only thing that will give the game away is inspecting the Received: headers, which will say it came from some random IP with no listed headers that say how it got from the mailing list provider to the random IP.

The spammer set up their mailing list to be so strictly anti-spam in order to deflect complaints submitted to the mailing list provider, especially more or less automatic ones created by people clicking on 'report this as spam' in their mail environment (which will often use headers put in by the mailing list provider). The mailing list provider will get the complaint(s) and hopefully not do much to the spammer overall, because all of the list members have fully confirmed subscriptions, a history of successful deliveries of past messages that look much like the latest one, and so on.

I don't know if any spammers are actively doing this, but I have recently seen at least one spammer that's doing something like it. Our mail system has logged a number of incoming (spam) messages with a null SMTP envelope sender that come from random IPs but that have valid DKIM signatures for various places. In some cases we have captured headers that suggest a pattern like this.

(You can also play this trick with major providers of free mail services; sign up, send email from them to some dummy mail address, and take advantage of the DKIM signature that they'll put on their outgoing messages. The abuse handling groups at those places will most likely take a look at the 'full' message headers and say 'it's obviously not from us', and they may not have the tools to even try to verify the DKIM signature to see that actually, it is from them.)

These days (and for a long time), most spam, phish, malware, and so on (in email and other things) is recognized not through general rules, patterns, and processes (eg), but by seeing if the content matches any known signatures. Sometimes this is literally matching cryptographic hashes, but more often there's some sort of signature matching engine involved with various matching operators, conditions for combining them, and so on. ClamAV is one example that's mostly a matching engine, which means that in practice you need a collection of signatures to make it useful. Since signatures aren't general things, they have to be created by someone and then you have to get that newly created (or perhaps updated) signature.

What people have collectively found is that in practice, the speed of updating signatures matters, often a lot; in fact it matters enough that people are willing to pay for faster updates to collections of signatures. Why it matters is pretty straightforward; you're in a race against attackers. Attackers are perfectly well aware that the effectiveness of what they're doing goes down fast once signatures are available for it (or in general once people have had time to recognize what's going on, get their web landing page killed off, or whatever), so they generally try to get things done as fast as possible.

(I'm sure there are some slow-moving spam, phish, and malware campaigns that keep on going and going, but I don't think they're very common.)

However, attackers have their own speed limits; they can only send so much so fast, to you and to everyone else. Against many attackers, this gives you the chance to cut off at least some of their activities if 'you' can react fast enough, which broadly means if you can get signature updates fast enough. In more sophisticated environments, fast signature updates may also give you the chance to re-scan people's recently received email messages before people open them (or when they open them).

(Similar things apply to scanning files or recognizing signs of active malware, especially since these may already be delayed from the initial attack depending on how the attacker got to people. If you're getting people to download malware from a web page by sending them a bait message, you have to wait for people to read their email.)

So in general, the faster you get signature updates, the less you'll be exposed to (and for a shorter amount of time). The slower the updates, the more you're exposed to and the longer you're exposed. In the extreme case, sufficiently delayed updates are mostly useless, because the attacker campaign they're reacting to is over by the time you get the updates active.

(Of course you can try to delay receiving things (and thus checking them), but this tends to be unpopular with people. Like it or not, modern email is expected to get through rapidly and as a result is used for time sensitive things.)

We've seen this ourselves when we changed from a commercial anti-spam system for our email to one mostly based on free software and free signature data sources for the anti-malware, anti-virus (and anti-phish) part. Even with paying for some signature sources, the free system clearly was less effective at matching and blocking new malware, and we're fairly certain that part of this was that the commercial system's signatures updated quite frequently (and the company involved had a bunch of people working on keeping them up to date).

(I think this is something that's well known to people in the communities that use signatures, like anti-spam and (anti-)malware, but is perhaps not so obvious to people outside those communities.)

TLS SNI (Server Name Indication) is a modern TLS feature where clients that are establishing a TLS session with a server tell it what name they are connecting to, so the server can give them the right TLS server certificate. TLS SNI is essential for the modern web's widespread HTTPS hosting, and so every usable HTTPS-capable web client uses SNI. However, other protocols also use TLS, and whether or not the software involved uses SNI is much more variable.

DANE is a way to bind TLS certificates to domain names through DNS and DNSSEC. In particular it can be used to authenticate the SMTP connections used to deliver email (RFC 7672). When you use DANE with TLS over SMTP, using SNI is required and is also straightforward, because DNSSEC and DANE have told you (the software trying to deliver email over SMTP) what server name to use.

Recently, SNI came up on the Exim mailing list, where I learned that when it's sending email, Exim doesn't normally use SNI when establishing TLS over SMTP (unless it's using DANE). According to Exim developers on the mailing list, the reasons for this include not being sure of what TLS SNI name to use and uncertainties over whether SMTP servers would malfunction if given SNI information. This caused me to go look at our (Exim) logs for our incoming mail gateway, where I noticed that although we don't use DANE and don't have DNSSEC, a number of organizations sending email to us were using SNI when they established their TLS sessions (helpfully, Exim logs this information). In fact, the SNI information logged is more interesting than I expected.

We have a straightforward inbound mail situation; our domains have a single DNS MX record to a specific host name that has a direct DNS A record (IP address). Despite that, a small number of senders supplied wild SNI names of 'dummy' (which look like mostly spammers), a RFC 1918 IP address (a sendnode.com host), and the IP address of the inbound mail gateway (from barracuda.com). However, most sending mailers that used SNI at all provided our inbound mail gateway's host name as the SNI name.

Using yesterday's logs because it's easy, roughly 40% of the accepted messages were sent using SNI; a better number is that about 46% of the messages that used TLS at all were using SNI (roughly 84% of the accepted incoming messages used TLS). One reason the percentage of SNI is so high is that a lot of the SNI sources are large, well known organizations (often ones with a lot invested in email), including amazonses.com, outlook.com, google.com, quora.com, uber.com, mimecast.com, statuspage.io, sendgrid.net, and mailgun.net.

Given this list of organizations that are willing to use SNI when talking to what is effectively a random server on the Internet with nothing particularly special about its DNS setup, my assumption is that today, sending SNI when you set up TLS over SMTP doesn't hurt delivery very much. At the same time, that some people's software send bogus values suggests that fumbling the SNI name doesn't do too much harm, which is often unlike the situation with HTTPS.

PS: I suspect that the software setting 'dummy' as the SNI name isn't actually mail software, but is instead some dedicated spam sending software that's using a TLS library that has a default SNI name set and is (of course) not overriding the name, much as some web spider software doesn't specifically set the HTTP User-Agent and so inherits whatever vague User-Agent their HTTP library defaults to.

It is probably not news to anyone reading this that some of the time, spammers sending you email will forge the email as being from various addresses at your domain, for either or both of the SMTP 'MAIL FROM' envelope sender address and the From: header address. Spammers have been doing this to us for years. What I hadn't realized until now, when I looked at the actual addresses being forged, was that spammers were forging various variations on 'noreply@<us>', in various variations of words and cases. Over the past ten days we've seen all of 'noreply@', 'Noreply@', 'Nonreply@', 'no_reply@', 'NOREPLY@', 'no-reply@', and 'NO-REPLY@'.

Of course, spammers also forge various plausible administrative addresses as well, such as 'Administrator@', 'Admin@', 'cpanel@', 'support@' (and 'Support@'), and one case of 'hr@', as well as the expected 'postmaster@'. These are almost all addresses that don't exist here and never have, so I'm pretty confident that spammers are just making them up instead of drawing them from a list of (past) legitimate email addresses of people here. I suspect that some or perhaps many of these forged addresses are being used on phish spams, and this is probably the case for the various 'noreply@' addresses.

(Spammers clearly use old email address lists to generate their envelope sender addresses, because we reject a lot lot of SMTP 'MAIL FROM' addresses that used to be real email addresses here but which have since been removed (we do eventually close some accounts). Interestingly, there is also a relatively frequently forged sender address that is a single-letter typo for a real person's email address.)

One of the lessons I draw from this little exercise in curiosity is that if we've created administrative-like email addresses in our system simply to reserve them, and we aren't using them, we should actively block their use as external sender addresses. If we want to create a dummy 'cpanel@' address, for example, we should definitely make it so that it's not accepted as a SMTP envelope sender.

(Because of some features of our mail environment, people here can created valid email addresses without our involvement (this has various entirely legitimate uses, including expendable personal email addresses). Historically this has meant that we grabbed a number of addresses simply as precautions to reserve them, without ever intending them to be 'legitimate'.)

PS: We do have a local noreply-like address, for internal use. However, spammers don't seem to forge it on their messages, perhaps because it basically never appears on email we send to actual people and thus has never made it onto various spammer lists of email addresses here.

(All of the email that we send to people has real sender and reply addresses that are read by us, even if the mail is sent by automated systems.)