Introducing "the Dip"

Our presentation at the recent MAAWG meetings focused on the effectiveness of Inbound Traffic Control in dealing with spam from unknown senders that represent most of drops seen in anti-spam effectiveness.

Two parts of the presentation really stood out with the audience, the second was a look at what a 98% capture-rate really means to an anti-spam lab.

Introducing "the Dip"



Despite 98% long term capture rates leading anti-spam systems experience significant drops in effectiveness when both sender and content are unknown, the most common times being the use of botnets, targeted campaigns not passing through a central lab and new spam approaches.

Any anti-spam lab worth its salt has a display that looks something like this graph in their lab showing their capture rate over time. Most of the time the capture rate is acceptably high, but once in a while – typically several times a day – the spam starts flooding through and then it’s all hands on deck while the lab figures out where that mail is coming from and how to plug the dike this time.
Sometimes the fix is elegant and long lasting, and sometimes its not.
The new technique can be network oriented or content oriented, and in either case the dip is what results.


From an end users and a service provider’s perspective you can flip this curve upside down and the dips become peak traffic loads, spam outbreaks, help desk calls and flooded inboxes.

Dips happen because anti-spam companies cannot have perfect insight into the spamming world.

• It takes enormous visibility and time to turn a new attack into the actionable quantities of known content and known senders.
• It takes the best filters 10 minutes to widely deploy a new filter rule capable of really making a dent in a new spam campaign.
• The blacklists take between 15 and 30 minutes to set up and distribute a new IP block.

Wouldn’t it be great if we could make the unknown senders wait around for a while – at least until we’ve had a chance to set up a filter rule?

In fact we can, this is one of the benefits of Inbound Traffic Control, messages from unknown senders are forced to wait for better anti-spam information. Taking away the spammers head start.

Spammers are Less Patient than Legitimate Senders

Our presentation at the recent MAAWG meetings (Messaging Anti-Abuse Working Group, 12th General Meeting Feb 18-20, 2008, San Francisco, California) focused on the effectiveness of Inbound Traffic Control in dealing with spam from unknown senders that represent most of drops seen in anti-spam effectiveness.

Based on discussions afterward, two parts of the presentation really stood out with the audience, the first was the difference in spammer and legitimate sender behavior when faced with a slow connection.

Spammers as Less Patient than Legitimate Senders



What this graph shows is that spamming MTA’s are less patient than legitimate senders. The economics of spam means that if forced to wait by a slow MTA the spammers will abandon the connection, usually within 10 seconds and move on while legitimate senders will wait to complete message delivery.

• The SMTP RFC recommends that email servers wait at least three minutes for each chunk of data they send to be received by the receiving server and acknowledged via a TCP acknowledgment packet.
• Furthermore, the RFC recommends that senders wait at least ten minutes for the final message delivery acknowledgment.
• These long timeouts were established because in the early days of the Internet, the infrastructure was slow and unreliable, and the machines were easily overloaded, leading to frequent message delivery delays.
• Today, email servers and our networks are much faster, processing incoming messages in a matter of seconds. Delays still occur, but the timeouts defined in the RFC are vastly higher than what is required in today's world.
• For what we imagine are economic reasons, spammers set their SMTP timeouts on the order of seconds rather than the levels recommended within the RFCs.
  

This graph compares the timeouts for spam traffic versus legitimate traffic. 90% of spam connections are gone within the first 10 seconds, whereas legitimate senders hang on for at least a couple of minutes.

The gap between these two lines is one of the things that Inbound Traffic Control can help you to exploit to reduce spam levels.




Ken Simpson talking at MAAWG

For those of you fortunate enough to be attending February's MAAWG (Messaging Anti-Abuse Working Group) conference in sunny San Francisco, I will be speaking on "why 98% accurate doesn't really mean 98% accurate." Spam filters have gotten pretty good "on average." The spam war is now all about dealing with those gaps in filter effectiveness and reputation system coverage that occur from time to time, showering end users with bursts of spam. This problem of "dips" is particularly acute for service providers, who suffer mass defections of users when filter accuracy dips.

I will be giving my talk on Tuesday February 19 at 2:30pm at the Mark Hopkins hotel on Nobb Hill. Details are on the MAAWG agenda for those of you who are members of the organization. Pre-talk suggestions are welcome - get in touch with me by leaving a comment.