Engineering Using Transport Layer Security (TLS) With a Transparent SMTP Proxy By Ken Simpson | 3 minute read Transport Layer Security (TLS) is an encryption protocol that ensures the privacy of SMTP traffic exchanged between mail servers. When TLS is enabled on an SMTP connection, an intermediary SMTP filtering proxy such as MailChannels Dedicated cannot decrypt and inspect the traffic without taking one of three possible actions, which are described in this article. Option 1: Prevent TLS Encryption by filtering the STARTTLS capability The TLS protocol is initiated by an SMTP client only if the SMTP server advertises the availability of TLS by responding to the EHLO command with the STARTTLS extended SMTP capability. For example, Google’s mail server responds to EHLO as follows: 250-SIZE 157286400250-8BITMIME250-STARTTLS250-ENHANCEDSTATUSCODES250-PIPELINING250-CHUNKING250 SMTPUTF8 One way to prevent TLS encryption by the client therefore is to block the STARTTLS capability from being advertised to the client. If STARTTLS is blocked (or filtered out) by the SMTP proxy, then Gmail’s response to EHLO would look like this to the SMTP client: 250-SIZE 157286400250-8BITMIME250-ENHANCEDSTATUSCODES250-PIPELINING250-CHUNKING250 SMTPUTF8 Option 2: Offer TLS to the client using a certificate hosted on the proxy The second option is to have the SMTP proxy offer its own TLS encryption to the client using a certificate stored on the proxy. This method allows for the session between the client and the proxy to be encrypted; however, the downside is that the client will not see a certificate that is a valid match with the server it intended to communicate with. Some SMTP clients will detect the certificate mismatch and will refuse to establish a connection; however, in our experience, this situation is extremely rare. If the SMTP proxy provides its own encrypted session with the SMTP client, then it’s crucial that the proxy also use TLS with the downstream mail server to which the client was trying to connect. That connection can be opportunistically secured with TLS by the proxy and the proxy should use best practices to verify that the certificate of the downstream server is valid for the purpose of maintaining connection security and privacy. Option 3: Pass through TLS connections without inspection The final option – which perhaps isn’t really an option – is to ignore TLS entirely, simply passing TLS encrypted session data between the client and the downstream SMTP server without any inspection. This option obviously does not allow the proxy to block spam that might be sent by the SMTP client, because the proxy can’t see the spam. However, it does respect the privacy of the SMTP session between the client and server and may be the only option available in situations where the service provider does not have the legal authority to inspect encrypted traffic. Conclusion Dealing with TLS encryption for transparent outbound spam filtering requires some decision-making on the part of the provider. Which option above is right for your network depends on the relationship you have with your clients, and the legal situation in your jurisdiction. MailChannels commissioned a report on the privacy implications of transparent SMTP filtering within the European Union, which we recommend as reading material for all providers – even those not located in the EU – because it provides great general advice on this topic. This report, and other resources, can be downloaded here.