Post #1 on Why Spam Filters Suck "trickle blog" series

A Short History of Spam Protection

While methods have changed, spam continues to be the misuse of an open communication network for financial gain. What was once a harmless annoyance has led to serious conditions where high spam traffic can clog email servers to the detriment of legitimate mail.

How did we get here? And what can we change to solve the problem?

The first spam email ever was used to promote a seminar from Digital Equipment Corporation (DEC) in 1978. I'd call it spam because it was a mass emailing harvested from a printed directory of ARPAnet to recipients who had not requested any contact.

Spam didn't become a huge problem until around 2002 when there were enough active email users worldwide to make spamming profitable. In response, the first commercial and open source spam filters arrived in Brightmail, PureMessage, and SpamAssassin to name a few. The first generation of filters applied sets of rules to each message received, identifying features within messages which might indicate the likelihood
of being spam.

Spammers countered rule-based filters by obfuscating the content of their messages. Rather than sending a text message advertising Viagra, for example, the spammer might chop the message into small HTML pieces which, while unrecognizable to the spam filter, would still render into legible text for the message recipient. The rule-based filters added more rules to catch these obfuscations, causing the spammers to further innovate. This pattern of content obfuscation continues to the present day, the most recent example of which is probably MP3 spam (i.e. spam message contained in an audio file).

Anti-spam is one of those areas of IT where you're "damned if you don't." If email is flowing free of spam, you hear nothing. But when spam is getting through or emails are backlogged on the server, there's hell to pay.

Why is spam causing backlogs? Why is all mail treated equally? And do we need to keep adding what are effectively junk processing servers?

As the sophistication of spam has increased so has the need for processing power to analyze those messages. Today, with email servers under high traffic loads, the ever increasing computational cost and processing overhead of analyzing the content of every email often results in service disruptions for legitimate email. This has to change. IT infrastructure costs should be a function of legitimate activity not spammer driven loads.

To solve the loading problem imposed by the current method of spam filtering where all incoming email messages are accepted by the server, buffered in a common queue on a first-come first-served basis, there needs to be a shift away from a single-queue of email traffic towards a prioritized system that can expedite legitimate mail first.

But there's more that needs to be considered...

UPDATE: On the subject of the history of spam, Christopher Nickson writes that the word "spam" to describe unsolicited commercial email recently celebrated it's 15th anniversary.

NEXT: Post #2 Prohibition Induces "Botlegging"

Irish Stock Exchange Website Compromised

I regularly read the US-CERT (United States - Computer Emergency Readiness Team) website to follow current activity. This morning, there was a warning of an attack that has compromised a large number of legitimate websites. One of the compromised sites generated quite a bit of media attention since it's a security related website. However, I thought it might be interesting to see if any other well known websites had also been compromised.



After a little searching I saw that the Irish Stock Exchange website, www.ise.ie had been a victim of the attack. The Announcement List asp page had been compromised in an attempt to add a link to a malicious Trojan hosted on a different web server. Fortunately, this issue has already been resolved but evidence of the attack against the site is revealed in a google cache search of the announcement page.

Is MailChannels queuing Yahoo mail?

It should be known that we are not currently working with Yahoo! Inc. However, we have heard murmurings that they have enforced more stringent filtering policies in addition to their own home-grown traffic shaping solution.

After Ken Simpson’s MAAWG presentation, we were contacted by a concerned system administrator who wanted to find out if our email traffic shaping has something to do with Yahoo’s current issue of widespread queuing and backlogs when sending mail to Yahoo! Mail users. Postmaster mailing lists have been reporting two situations:

1. Yahoo is returning 421 Messages, temporarily deferring every message attempt. Apparently Yahoo never accepts the message, even after a day of retrying. Whether you use DomainKeys or SPF records, Yahoo is deferring the message as soon as your mail server connects, so it never gets a chance to see the header.
2. Good email are not being bounced, but tagged as Spam and filed directly to the Spam folder that Yahoo users rarely check. As worldwide spam volume rises, users become numb to the ballooning size of quarantines and check Spam folders less often.

Needless to say, this filtering policy equates to bad service for Yahoo users. Whether or not they also multiplex the 'shaped' connections is not known, but this would be a good time to shed some light on the topic of email queuing.

SMTP typically works in a queue where mail is delivered in the order it arrives. In most cases, administrators don’t know when and for how long their mail is being delayed by backlogged mail servers. But relying on the fact that “mail will get delivered whenever it can” is a bit of a cop out.

Queuing email on the receiving server is an unnecessary artifact of the history of slow email servers with limited connections. It’s better to just keep all the connections alive and process them according to priority. But queuing makes this difficult because receiving the message into the queue in the first place is done in the order of receipt not by priority. Then once the message is in the queue it is now the responsibility of the recipient despite the fact it has not yet been processed and delivered.

By deploying traffic shaping, good senders are no longer forced to retry when connection limits are hit because the connections are maintained. If Yahoo were to deploy reputation-based prioritization, only bad senders and unknown senders would be penalized. Traffic shaping prioritizes email on servers to reduce the delay and loss of non-spam emails due to queuing. This is not to be confused for anti-spam vendors who rate-limit on quantity of emails, rather than the speed of email delivery.

Traffic shaping on SMTP is often mistaken for connection rate-limiting. Rate limiting sets a quota on the maximum amount of messages a spamming source can send in a given time, i.e. 6 messages per minute. After which, all future messages are rejected. On the other hand, traffic shaping involves interfering with network traffic by reducing the bandwidth for SMTP connections. It performs and implements similar to a network load-balancing device to pool and reduce overall email volume.

Were Yahoo to be doing SMTP traffic shaping, no messages would be lost or deferred. In fact, traffic shaping prioritizes sources with good traffic and throttles sources that are sending spam, resulting in up to 70 per cent reduction in overall email volume.