Is Email Flawed By Design?

Last week Sarah Palin's personal email account being hacked raised concerns about webmail security. Although we covered this in our hugely successful trickle blog earlier this year, it might be worth talking about how things have gone wrong.

1. SPAM FILTERING ONLY SEES IN BLACK AND WHITE

We've become used to receiving email within seconds. If you know somebody sent you a message, you expect it in your inbox immediately but that is not always the case. The problem is that spammers and anti-spam systems are constantly negotiating for your server's attention.

As anti-spam systems get more accurate, spammers have to send more to improve their chances of getting through to us. Blocking spam with a hard pass or fail system doesn't make sense for new, but untrusted senders in the grey area.

2. YOUR SERVERS BECOME FLOODED

Blocking separates spam from good email. High connection volume is harder to defend against and can take a damaging toll on your network causing deliver delays for hours sometimes. The problem is a typical email system works in a queue. When spam traffic is high enough, it clogs your network and crowds out good mail.

For businesses hosting their own on-premise email servers, filtering everything is incredibly expensive because:

a) You have to add more server hardware to keep up the CPU required to process all that mail

b) IT staff are dealing with spam maintenance when they could be working on more proactive projects

c) Support complaints pile up when spam outbreaks reach hosting customer inboxes

d) Overloaded servers impact the timely delivery of good email

3. EMAIL LACKS PRIORITIZATION

Email is processed in the order it arrives, on a first come first served basis. Imagine if you had to do every task in your day in the order it arrived. This gives you an idea of how email servers work. When spam clogs servers, the mail queue is too busy analyzing spam meanwhile good mail is forced to wait. Wouldn't it be nice if good email had a special lane expedited for delivery and skip lengthy queues?

4. SPAMMERS HAVE THE ADVANTAGE

When a new spam outbreak happens, it can take 10-30 minutes for spam labs to identify and distribute updates to filters. Enough time for spam to flood servers. Most people can drop a lot of the bad senders using DNS blacklists and reputation filters drop a lot of the spam but it takes minutes or hours to update DNSBL's to block the latest botnets.

Whether you're managing Sendmail, Postfix or another mail server, email traffic shaping reduces your email volume by 90% or more and pools your mail server to handle more than 25,000 concurrent connections.

We'd love to hear your comments. Let us know your biggest gripe about 'email.'

Post #10 on Why Spam Filters Suck "trickle blog" series

Challenges with Throttling

Slowing traffic from spammers works well to decrease volume and contain infrastructure costs. It allows you to deal effectively with the large proportion of senders that are not yet listed in any blacklist by making spammers give up. The problem with slowing down spammers is that it increases the number of TCP connections to your email server.

One customer dealt with 100 connections at a time, but after traffic shaping, now sees upward of 1000 concurrent connections. This ten-fold increase in number of connections utterly destroys most email servers. To illustrate this problem, consider that it takes between two seconds to deliver an email message under normal circumstances. Slowing down a spam zombie causes the connection to last an average of 40 seconds. If a significant proportion of connections are lasting 30 times longer than normal, then the number of connections you have going on at any one time grows.



The graph above shows the number of SMTP connections being handled by a single server at a large university in the New York area. Noteworthy is that the number of concurrent connections hovers around 500. The red line represents the total number of connections. The green line indicates the number of connections that our traffic shaping is choosing to slow down.

Administrators running Sendmail or Postfix will note that 500 concurrent connections is a large number. The amount of memory required to handle 500 concurrent Sendmail processes, plus any associated spam filtering processes, is considerable. If we were passing this number of connections through to Sendmail, the email server would almost certainly become overloaded.

One approach to improve the scalability of email systems is to completely redesign your email server with a new highly scalable software architecture. But re-designing the email server is difficult, and changing the email system is a large commitment. An asymmetric SMTP proxy that we call real-time SMTP Multiplexing is designed to solve the scalability challenge posed by traffic shaping. The proxy accepts thousands of connections from the Internet and then multiplexes these connections onto a much smaller pool of connections with the existing email server. Unlike an email server, the Multiplexing proxy doesn't save messages to disk, which means it is a lot less complex and also doesn't consume much in the way of system resources.



This graph shows the number of connections to the email server of the large university mentioned previously. The red line indicates that the average number of connections with the email server hovers around 50, which is well within the amount a typical email server can handle. By multiplexing the SMTP connections, the system can achieve a 5:1 or 10:1 reduction in the number of connections the email server has to deal with. Moreover, reducing the concurrency of connections the email server has to deal with, enables the ability to reduce a large proportion of the incoming connections, getting rid of a great deal of spam traffic in the process.

To get updates, subscribe to the RSS feed (unsubscribe at any time).

Related Posts:

• Real World Scenarios
• Dealing Spammers a Blow
• Slowing Spammers Down
• Blocking Spam in 2008
• Why Botnets are so Difficult to Stop

Google Apps are Taking Off?

Some of you may be familiar with MailChannels' "PingedIn" service. Every night, we survey the mail servers of approximately half a million companies worldwide, using a proprietary algorithm to determine the kind of email server software they are using to receive email.

Recently I was reviewing historical data stretching back to mid-summer, when I noticed a strong trend:


The lime green line shows that there has been a 50% increase in the number of companies using Google to host their email. This is a really impressive rate of growth in what has been a fairly stagnant industry for the past few years.

Other interesting observations:

• The decline of software: more and more companies are outsourcing their edge email solution to someone else. The only exception we found to this rule was MXLogic, who appear to have lost about 5% of their customers since mid-summer (according to our data -- please don't sue us).
• The flattening of IronPort: There has been virtually no growth at all in IronPort's installed base since they were acquired by Cisco. That said, at least they haven't lost ground.
• Continuing high rate of churn: Not shown on the graph, but tracked by PingedIn is the rate at which companies move from one solution to another. We are continuing to see an approximately 20% annual churn rate in the email boundary market.

Now, even though Google's growth may appear spectacular, it should be taken with a grain of salt. There are more than 17,000 sites in our database running Barracuda email appliances. Google still has just 1800. But at their current rate of growth, Google should surpass Barracuda some time in 2010. By the same token, even though there are still 42,000 Sendmail sites in our database, at its current rate of decline, Sendmail will be all but extinct in 2015.

Okay, 2015 is a really long way away. Sendmail is going to be with us until the end of time.

Does Sendmail Throttle?

The MailChannels TrafficControl product reduces spam by throttling the sender so that a spammer will eventually give up. If I explain this to a system administrator it's commonly met with the question "Doesn't Sendmail already Throttle?"

I can understand where the confusion arises since the term "throttle" is quite generic and can be applied to any type of restriction or limiting to a resource. So I thought it might be worthwhile explaining things further.

In the case of TrafficControl the entire SMTP conversation is drastically slowed down in both directions to emulate a slow connection. This means that a never before seen zombie sending an image spam e-mail of around 20kB could expect to transfer data at a rate of 15 bytes per second (configurable) which could mean having to hang around for 20 minutes or so. Since the system is designed with SMTP multi-plexing in mind from the ground up it can handle thousands of concurrent connections and legitimate connections are given a fast track to the MTA since they have a good reputation.

In the case of Sendmail, it's a much different story. In the situation described above, the message from the zombie sender would be delivered in the same way as legitimate mail. The difference is that TrafficControl limits the data rate and Sendmail limits the connection rate which are two very different things. So what exactly does Sendmail throttling do? The following rate limiting features are explained:

ratecontrol - provides a per minute connection rate window that limits the number of connections, usually from a single server. If a connecting server exceeds this quota, sendmail then returns a 4xx response indicating it is temporarily unavailable. This is somewhat useful during a Denial of Service attack from a single machine but not useful against a distributed attack.

conncontrol - whereas ratecontrol limits the rate of connections, it's still possible for an attacker to gradually build up a pool of connections with Sendmail and eventually exhaust it's resources. The conncontrol feature therefore provides a limit on the total number of concurrent connections an individual host is allowed to maintain.

ConnectionRateThrottle - is similar to the ratecontrol feature but instead it provides a global per minute connection rate window that isn't specific to an individual host. This is more useful in the case of a distributed attack. However, it's still possible for the machines to continually build up connections over a long period of time and exhaust resourcess without hitting this quota.

greet_pause - as the name suggests is the addition of a brief pause in milliseconds before sending the banner greeting message. If a sender attempts to send additional commands without listening for the greeting the connection can be marked bad. This is easily circumvented by most spammers using sending software that actually listens for SMTP replies.

I hope this helps explain the difference between rate limiting connections versus slowing down an entire SMTP conversation to force a spammer to quit. If you'd like more information you can read the following description.