Is Email Flawed By Design?

Last week Sarah Palin's personal email account being hacked raised concerns about webmail security. Although we covered this in our hugely successful trickle blog earlier this year, it might be worth talking about how things have gone wrong.

1. SPAM FILTERING ONLY SEES IN BLACK AND WHITE

We've become used to receiving email within seconds. If you know somebody sent you a message, you expect it in your inbox immediately but that is not always the case. The problem is that spammers and anti-spam systems are constantly negotiating for your server's attention.

As anti-spam systems get more accurate, spammers have to send more to improve their chances of getting through to us. Blocking spam with a hard pass or fail system doesn't make sense for new, but untrusted senders in the grey area.

2. YOUR SERVERS BECOME FLOODED

Blocking separates spam from good email. High connection volume is harder to defend against and can take a damaging toll on your network causing deliver delays for hours sometimes. The problem is a typical email system works in a queue. When spam traffic is high enough, it clogs your network and crowds out good mail.

For businesses hosting their own on-premise email servers, filtering everything is incredibly expensive because:

a) You have to add more server hardware to keep up the CPU required to process all that mail

b) IT staff are dealing with spam maintenance when they could be working on more proactive projects

c) Support complaints pile up when spam outbreaks reach hosting customer inboxes

d) Overloaded servers impact the timely delivery of good email

3. EMAIL LACKS PRIORITIZATION

Email is processed in the order it arrives, on a first come first served basis. Imagine if you had to do every task in your day in the order it arrived. This gives you an idea of how email servers work. When spam clogs servers, the mail queue is too busy analyzing spam meanwhile good mail is forced to wait. Wouldn't it be nice if good email had a special lane expedited for delivery and skip lengthy queues?

4. SPAMMERS HAVE THE ADVANTAGE

When a new spam outbreak happens, it can take 10-30 minutes for spam labs to identify and distribute updates to filters. Enough time for spam to flood servers. Most people can drop a lot of the bad senders using DNS blacklists and reputation filters drop a lot of the spam but it takes minutes or hours to update DNSBL's to block the latest botnets.

Whether you're managing Sendmail, Postfix or another mail server, email traffic shaping reduces your email volume by 90% or more and pools your mail server to handle more than 25,000 concurrent connections.

We'd love to hear your comments. Let us know your biggest gripe about 'email.'

Is MailChannels queuing Yahoo mail?

It should be known that we are not currently working with Yahoo! Inc. However, we have heard murmurings that they have enforced more stringent filtering policies in addition to their own home-grown traffic shaping solution.

After Ken Simpson’s MAAWG presentation, we were contacted by a concerned system administrator who wanted to find out if our email traffic shaping has something to do with Yahoo’s current issue of widespread queuing and backlogs when sending mail to Yahoo! Mail users. Postmaster mailing lists have been reporting two situations:

1. Yahoo is returning 421 Messages, temporarily deferring every message attempt. Apparently Yahoo never accepts the message, even after a day of retrying. Whether you use DomainKeys or SPF records, Yahoo is deferring the message as soon as your mail server connects, so it never gets a chance to see the header.
2. Good email are not being bounced, but tagged as Spam and filed directly to the Spam folder that Yahoo users rarely check. As worldwide spam volume rises, users become numb to the ballooning size of quarantines and check Spam folders less often.

Needless to say, this filtering policy equates to bad service for Yahoo users. Whether or not they also multiplex the 'shaped' connections is not known, but this would be a good time to shed some light on the topic of email queuing.

SMTP typically works in a queue where mail is delivered in the order it arrives. In most cases, administrators don’t know when and for how long their mail is being delayed by backlogged mail servers. But relying on the fact that “mail will get delivered whenever it can” is a bit of a cop out.

Queuing email on the receiving server is an unnecessary artifact of the history of slow email servers with limited connections. It’s better to just keep all the connections alive and process them according to priority. But queuing makes this difficult because receiving the message into the queue in the first place is done in the order of receipt not by priority. Then once the message is in the queue it is now the responsibility of the recipient despite the fact it has not yet been processed and delivered.

By deploying traffic shaping, good senders are no longer forced to retry when connection limits are hit because the connections are maintained. If Yahoo were to deploy reputation-based prioritization, only bad senders and unknown senders would be penalized. Traffic shaping prioritizes email on servers to reduce the delay and loss of non-spam emails due to queuing. This is not to be confused for anti-spam vendors who rate-limit on quantity of emails, rather than the speed of email delivery.

Traffic shaping on SMTP is often mistaken for connection rate-limiting. Rate limiting sets a quota on the maximum amount of messages a spamming source can send in a given time, i.e. 6 messages per minute. After which, all future messages are rejected. On the other hand, traffic shaping involves interfering with network traffic by reducing the bandwidth for SMTP connections. It performs and implements similar to a network load-balancing device to pool and reduce overall email volume.

Were Yahoo to be doing SMTP traffic shaping, no messages would be lost or deferred. In fact, traffic shaping prioritizes sources with good traffic and throttles sources that are sending spam, resulting in up to 70 per cent reduction in overall email volume.