Update: How well do large senders do at delivering email?

Update: Two weeks ago, we posted an article discussing how successfully the largest email senders are able to reach the inbox. We received some criticism about this post, because it made some providers look worse than they really ought to be based on their own inbox delivery performance metrics.

To help shed some light on the subject, we redid our analysis, including 7x more source data, and looking under the hood a bit to find out why and when messages are being rejected from each of these large senders. It turns out that many large senders do very well at having messages accepted once their SMTP connections reach the DATA phase; however, they have very different rates of acceptance at earlier phases in the SMTP connection sequence.

As a primer, the SMTP protocol has several “phases,” and in each phase, the sender transmits an additional quantum of information that helps the receiver route messages. Here are the SMTP phases (and I know I’m skipping some for the more technical people in our audience):

• Connect – this phase happens immediately after the SMTP client connects with the server. The server’s job at this phase is to assess whether it wishes to accept any further communication from the client’s IP address.
• MAIL FROM – the client supplies the sender’s email address.
• RCPT TO – the client supplies the email addresses of each message recipient.
• DATA – the client provides the actual message content to be delivered to the recipients.

Email can be rejected at any SMTP phase by an email server. Additionally, there are many types of rejections (and our own Response Analytics™ technology sorts these types of responses for our outbound spam filtering customers). Being rejected at a particular phase does not mean that the receiver suspects a message is spam; rejections can happen for a huge number of reasons.

In the table below, we summarize what proportion of SMTP connections originating from each provider are rejected at each SMTP phase. For example, Exact Target (exacttarget.com) has 1.5% of its connections rejected at the Connect phase, 0.1% at MAIL FROM, and just 0.4% at DATA. Machines sending from Microsoft’s network (microsoft.com) are rejected only 0.4% of the time at Connect, but 4.1% of the time at DATA. These statistics might imply that Exact Target sends message content that is more desirable for receivers than Microsoft (whose DATA reject rate is higher).

Keep in mind that the statistics here are for ALL SMTP traffic that we sampled from the sender’s entire network over a one week period. For some providers who offer a large variety of services such as cloud hosting, we can expect their numbers to look worse than other providers who specialize only in email sending.

Another important thing to keep in mind is that rejections in the early SMTP phases (Connect & MAIL FROM) DO NOT imply that the sender is doing anything wrong. It could just mean that the sender is being rate limited, and that the rate limiting is occuring at the Connect phase, causing many connections to drop.

If you are interested in how well a sender does at ensuring their traffic is free of spam, the best place to look is the DATA column. This column indicates what fraction of messages are rejected inline for any reason; however, the most common reason for rejecting a message is because it has been deemed to contain spam content.

Analysis

Some of the sending networks appear to be losing a lot of SMTP connections at Connect time. Our best guess is that these rejections are occurring because of rate limiting techniques being used by the receiver. While being rate limited does not necessarily imply that a sender is doing anything wrong, adjusting delivery speed to avoid being rate limited may present an opportunity to reduce the impact on the receiver’s network. Other networks are losing connections because of IP blocking. We have chosen not to break out that segment of our data in this analysis, however may do so in future.

At the RCPT phase, two sender networks seem to lose some connections. The worst is twtelecom.com (not to be confused with Time Warner Telecom), which loses 2.6% of its SMTP sessions at RCPT. SendGrid loses 0.4%. Connections are generally lost at RCPT time because of failed recipient validations. This could indicate that the sender needs to trim their recipient lists to remove invalid recipients.

Finally, at DATA, we see that most of the networks that aim to send email for a business do very very well at delivering at this stage. Between Constant Contact, MailChimp, SendGrid, and Exact Target, the connection failure rate at DATA is well below 1%. In fact, the only networks that seem to struggle at the DATA phase are those providing hosting services: Microsoft (whose network includes the Azure cloud – not just email services like Live Mail), Coloquest, and as61440 – otherwise known as Digital Energy Technologies or Host1Plus, all seem to fall down at the DATA phase.

Caveats

Our data set is not even close to comprehensive or error free, and you should not make purchasing decisions based on this blog post if you are looking for a network to send your email for you. That said, we welcome your questions and feedback in the comments below.