SPF-aware greylisting and filter-greylist · poolp.org

[poolpOrg]

[jogness]

I am happy to see you talk about SPF-aware greylisting. This seems like the natural solution.

Also, I am also happy to see you talk about tarpitting via laggy SMTP transactions. On my exim4 servers I do this using the delay control for the deny statement. I was a bit disappointed that opensmtpd doesn't have that. I was expecting something like:

match <options> delay 100s reject

... or maybe it does have something for that, but I haven't found it yet. I am new to opensmtpd but so far am quite happy with it. Thank you for your great work!

[poolpOrg]

Thanks for your comment :-)

I have committed an additional change to filter-greylist today which whitelists the domain of RCPT addresses when the session originates from a unix socket or has been successfully authenticated. This automatically whitelisting domains that a trusted sender has emitted mail to.

For the tarpitting, we decided not to bring that in builtin filters because it is tricky to do there and it is trivial to do in proc filters. I have written filter-senderscore which looks the reputation of a client IP address in the SenderScore database and allows blocking or junking content if reputation falls below a certain threshold. In that filter, I have a -delayFactor option which introduces a delay inversely proportional to the reputation causing high reputation IP to have no delay and low reputation IP to be essentially tarpitted.

[estrogently]

Thank you for your work on all these filters and on OpenSMTPd in general!

I've been noticing some connections from Google servers still getting greylisted by IP rather than domain, and thus never getting whitelisted. My guess is these are from G Suite domains which haven't (correctly) set up SPF. I'm not sure what to do about that.

[poolpOrg]

@estrogently could be that yes, without having an example domain it's hard to tell.

There's not much to do with domains with invalid / missing SPF anyways, a good way to handle them in my opinion is to Junk them by default so they end up in a Spam folder then manually whitelisting them if they're worth it. They're going to be penalized by the big mailers so penalizing them too is just aligning with the industry at this point.

[myfirstnameispaul]

@poolpOrg I have worked for old small companies that registered their domains in the late 90s to early 00s. They never had any SPF record but they inbox everywhere. There must be something more to reputation, even to the oligopolists. They even switch mail providers and the reputation follows them.

[poolpOrg]

@myfirstnameispaul that is not surprising.

Domains have a reputation that goes beyond just technical details like having setup SPF, DKIM and DMARC. The reputation is not just a technical validation, it is tied to the likelihood that recipients will complain when the domain sends them mail.

Let's take Gmail for example: what they really care about is not so much that SPF, DKIM and DMARC are set for a domain, but that when a user expects mails from a domain to be inboxed, then these mails are really inboxed. No one wants mail from Amazon or Netflix in the spambox and I can guarantee you that either one of Amazon or Netflix could remove SPF, send you a mail saying "I WILL FUCKING SELL YOU VIAGRA BECAUSE YES THIS MAIL IS SPAM" and be inboxed with no issue. Their reputation and user expectation that these domains inbox is so high that they can do a LOT of things wrong and still be given a free pass. I'm using these extreme examples because the "domain X doesn't do Y so Y must not be that important" rationale is inaccurate: it's not because domain X has a reputation that's high enough to hide the effects of doing Y, that Y doesn't actually have an effect.

With that being said, big mail operators have all their own search engine, which means they all have an index of the Internet, know which domains are big domains, which ones are small but popular, which ones have existed for years, etc... If I worked at Google, I would definitely use the domain age and categorization to assign a reputation to a domain based on what the website does / sells / provides as a service, and if I have thought of that, I'm sure the smart people of Google, Microsoft, Yahoo and other also did. So it's not surprising that small companies that have had domains for ten or twenty years and selling things that people care about inbox regardless of not ticking all technical points, they have enough street cred to get some slack.

Then there is another aspect: reputation degrades with the number of errors you make. A domain may have enough street cred to compensate for a lot of errors, and it may keep increasing reputations so it can do more and more errors without impact. But a domain may also work for years and then have reputation shrink and no longer compensate errors, it's not unseen. Finally, you have the very small domains with no reputation whatsoever and any errors count so unless you tick the tech boxes right, your score falls below spambox almost immediately.

It is very important to always try to tick all tech boxes right because that is really putting yourself on the safest side, regardless of your reputation.