ICANN and IANA’s domains hijacked

I've been following the blog posts of security consultant Dancho Danchev for quite a while. I don't usually have an opportunity to mention his posts as this is an Anti-Spam blog and for the most part I try to keep my posts on topic. For example, a few weeks ago he mentioned a Comcast DNS hijack with some speculation of what may have happened. The attackers claimed they accessed Comcast's DNS account through a combination of Social Engineering and a technical hack. It sounds like a phone call, e-mail or fax could have been used to socially engineer some data. I'm not sure which technical hack was used but I'm hoping the registrars protect against brute force login attempts!

Following the attack, ICANN (Internet Corporation for Assigned Names and Numbers) the body responsible for managing the assignment of domain names and IP addresses, published a security advisory with recommendations on how to avoid these types of attacks. I took a very quick look at the document and found it a little strange - take the following paragraph:

The attacker can add or modify the following records in the domain zone data he controls:

• MX, to point to mail hosts under his control and use these to send spam. Using the registrant’s domain is preferable over domain the attacker could register directly because in many cases, the registrant’s domain is “trusted” by other mail systems; i.e., it has no history of originating or reputation for relaying spam and is not blacklisted or otherwise blocked from forwarding email.

Surely the most obvious attack would be to change the MX record to point to a mail host under the attackers control so that confidential inbound e-mail could be captured? I'm not entirely sure how they claim this would be used to send spam, perhaps in the case of a large e-mail provider where the outbound mailserver would use MX records to validate who can relay. However, for most companies this would not be the case.

It doesn't even mention the TXT record which could be another attack vector. The attacker could add one of their own IP addresses to the SPF record. This would allow them to forge the domain of the hijacked company and possibly bypass filtering if the domain is on a whitelist.

Embarrassingly, shortly after releasing the advisory ICANN was victim to such an attack. Both ICANN and IANA domains had their DNS compromised yesterday so they pointed to a different site. From an e-mail security perspective these attacks are quite scary - confidential e-mail could be accessed or very real phishing e-mails could be sent.

Day Zero Anti-Spam?

Many of you will already be familiar with the concept of a Day Zero Virus attack. Whenever a new vulnerability is discovered, it's likely that never-before-seen malware, without existing signatures, will start to appear. Given the danger of new attacks, AV vendors have developed various Day-Zero Anti-Virus solutions. For example, one e-mail security vendor delays messages with executable attachments for a number of hours to allow time for new AV signatures to be propagated.

The Anti-Virus companies are very aware that new virus campaigns will emerge, without signatures. They have solutions in place. However, in the world of Anti-Spam I don't hear much discussion of new spam campaigns and what companies are doing to help protect their customer base against these attacks. A dip in effectiveness occurs when a new spam campign is launched and filters are not yet in place to block it i.e. Day Zero Spam! In February, we discussed the idea of "The Dip" with regard to AS effectiveness and I thought it worth further discussion.

Anti-Spam rules can be pre-emptive or reactive. For example, heuristic rules look for generic spam indicators in a message that could catch a small percentage of spam e-mail from new campaigns. However, spammers can easily setup drop boxes at many ISP's to confirm successful delivery of the e-mail, before commencing the campaign. Reactive rules respond to active campaigns by creating targeted rules. Collected samples are required to write the rules against.

Typically, an Anti-Spam Operations center will have visibility into spam attacks via the use of honey pots to collect samples, as well as end user missed spam submissions. There's a delay in the spam sample being reported to the operations center as it may take some time for the end user to report it. Also, the honey pot may not detect the message until long after the campaign has commenced. As the number of submissions to the center is huge, there's a delay before the sample is prioritized to be processed by automated or human rule writers. Finally, after the rule has been created, there's a delay in propagating the rule set to customers.

The scenario above is an optimistic one. In some cases, it may not even be possible to create an effective rule that doesn't result in an increase in false positives. Think back to the crippling image spam attack over a year ago. So much legitimate corporate mail had images such as the company logo attached. It wasn't easy to create rules. Anti-Spam effectiveness took a hit. Another example could be a customer in the Middle East using a US-centric Anti-Spam product. The operations center may not have enough visibility into localized samples of spam appearing in Arabic or Hebrew. The same can be said for customers in Asia.

For the most part, Anti-Spam vendors seem to keep very tight lipped on these deficiencies. Earlier this week, Cloudmark announced their new ActiveFilter. I should mention that they're a partner of ours and we ship Traffic Control with Cloudmark. It's pretty neat in that it actually scans the message store until the message is retrieved to see if any messages subsequently receive a spam verdict. The interesting thing is that this was the first time I've heard a major player in the AS market openly discuss the problem with new spam attacks:

The messaging security landscape has always been an arms race between attackers and anti-spam providers. In an effort to penetrate the inbox and reach their target audience, spammers and hackers are deploying extremely sophisticated techniques to evade spam filters. A current trend is to use botnets to send out huge volumes of rapidly-changing messages as quickly as possible. These bots can send millions of messages in under a minute. Given the intensity and speed of attacks, it’s no surprise that spam now constitutes more than 95 percent of all e-mail traffic and even with the most effective e-mail filtering in place, a small amount of spam will still find its way into e-mail inboxes––these are the messages spammers are banking on.

I'd love to hear how other Anti-Spam vendors are dealing with Day-Zero Spam Attacks? In the case of Traffic Control, we throttle never-before-seen connections until they build up a good reputation. A sender is guilty until proven innocent. Traffic shaping is agnostic to the message content. It doesn't matter whether the spam message hides its content in images or Google Docs, or even if it is targeted in a language for a specific geographic region. I don't believe in a silver bullet to combat spam in the short term, but I do believe in a layered approach. Use Traffic Shaping up front to protect the MTA, and a good content filter to further reduce spam.

The Slowest Mail Servers in the World

MailChannels periodically surveys about a million mail servers around the world to find out what MTA software they are running. This survey is called PingedIn, and I have written about it previously (see the Network World article by Michael Osterman).

Recently, we enhanced PingedIn so that it keeps track of the amount of time it takes for the mail server we are connecting to to respond to each SMTP command. By analyzing these delays, we can make some very rough inferences about how heavily loaded a mail server might be relative to others in its class.

For fun, I summarized the slowest mail servers, looking at just two of the response times we track:

1. Banner Delay - this is the length of time between establishing a TCP connection with the mail server, and receiving the initial SMTP banner; and,
2. HELO Delay - this is the length of time it takes to receive a response to the HELO command (HELO with no arguments, BTW).

The banner delay does not necessarily indicate that the mail server is overloaded. Many service providers insert an intentional delay (called the "Greet Pause" by some), to hopefully deter spammers. Other providers use traffic shaping - MessageLabs is a good example - which slows down responses to all commands unless you are a trusted sender.

If the HELO delay is large, it more probably indicates a slow or overtaxed mail server. While many providers seem to provide an intentional banner delay, far fewer choose to delay subsequent commands. So if the HELO response is slow, it probably means the mail server is overloaded.

The following chart and table summarize the slowest overall mail servers and email services, based on the sum of their banner delay and HELO delay:



(click on the chart for a larger version)

The slowest entry is the Electric Mail Company (a division of j2 Global Communications), a service provider with a considerable base of mostly small business customers. I know that Electric Mail has always been a pioneer in using protocol-layer approaches to deter spam, so their long delays are likely intentional.

Next up is DirectNIC, a cheap domain registrar. DirectNIC uses Sendmail, and I think it's fair to guess that they have enabled the greet pause function to delay the banner, since their HELO delay is not too bad at only 1.2 seconds.

At 5.33 seconds, MXPath's HELO delay is also considerable. MXPath is a hosted email security service (by the looks of it, anyhow).

Interesting entries:

• Frontbridge - this is Microsoft's hosted email security service. At 3.2 seconds, their HELO response is 50% greater than average. The banner time is also high - are they using a greet pause, or is this indicative of performance issues?
• Symantec Norton Anti-Virus Gateway - Banner delay? Can anyone from Symantec comment?

If we set aside the banner delay and examine only the HELO delay, we arrive at the following chart:



The front-runner for HELO delay is, once again, Electric Mail. At 15.39 seconds, their HELO delay is more than seven times the average. Looks like an intentional greet pause.

More surprising, in my opinion, is the appearance of Google and Yahoo in the top ten HELO delays. Google has a large and flexible number of email servers, as does Yahoo. Are their HELO delays the result of traffic shaping, or high load?

What's your email server delay? Why not call us and we'll run a load test.

Post #10 on Why Spam Filters Suck "trickle blog" series

Challenges with Throttling

Slowing traffic from spammers works well to decrease volume and contain infrastructure costs. It allows you to deal effectively with the large proportion of senders that are not yet listed in any blacklist by making spammers give up. The problem with slowing down spammers is that it increases the number of TCP connections to your email server.

One customer dealt with 100 connections at a time, but after traffic shaping, now sees upward of 1000 concurrent connections. This ten-fold increase in number of connections utterly destroys most email servers. To illustrate this problem, consider that it takes between two seconds to deliver an email message under normal circumstances. Slowing down a spam zombie causes the connection to last an average of 40 seconds. If a significant proportion of connections are lasting 30 times longer than normal, then the number of connections you have going on at any one time grows.



The graph above shows the number of SMTP connections being handled by a single server at a large university in the New York area. Noteworthy is that the number of concurrent connections hovers around 500. The red line represents the total number of connections. The green line indicates the number of connections that our traffic shaping is choosing to slow down.

Administrators running Sendmail or Postfix will note that 500 concurrent connections is a large number. The amount of memory required to handle 500 concurrent Sendmail processes, plus any associated spam filtering processes, is considerable. If we were passing this number of connections through to Sendmail, the email server would almost certainly become overloaded.

One approach to improve the scalability of email systems is to completely redesign your email server with a new highly scalable software architecture. But re-designing the email server is difficult, and changing the email system is a large commitment. An asymmetric SMTP proxy that we call real-time SMTP Multiplexing is designed to solve the scalability challenge posed by traffic shaping. The proxy accepts thousands of connections from the Internet and then multiplexes these connections onto a much smaller pool of connections with the existing email server. Unlike an email server, the Multiplexing proxy doesn't save messages to disk, which means it is a lot less complex and also doesn't consume much in the way of system resources.



This graph shows the number of connections to the email server of the large university mentioned previously. The red line indicates that the average number of connections with the email server hovers around 50, which is well within the amount a typical email server can handle. By multiplexing the SMTP connections, the system can achieve a 5:1 or 10:1 reduction in the number of connections the email server has to deal with. Moreover, reducing the concurrency of connections the email server has to deal with, enables the ability to reduce a large proportion of the incoming connections, getting rid of a great deal of spam traffic in the process.

To get updates, subscribe to the RSS feed (unsubscribe at any time).

Related Posts:

• Real World Scenarios
• Dealing Spammers a Blow
• Slowing Spammers Down
• Blocking Spam in 2008
• Why Botnets are so Difficult to Stop

Storms and Earthquakes

The infamous Storm worm was named based on the Subject lines related to the storms across Europe at the time. For example, "230 dead as storm batters Europe" was commonly seen. Last year, we blogged about it and did a breakdown of infected machines by continent and country as well as discussing how it changes tactics.

This week, a new variant has appeared with Subject lines related to an earthquake in China. This is very similar to the social engineering tactics of the original campaign, where a natural disaster was used to entice the recipient to open the e-mail. The US Computer Emergency Readiness Team has posted an alert on it's website. The Subject lines currently being seen in the e-mails are as follows:

* The most powerful quake hits China

* Countless victims of earthquake in China

* Death toll in China is growing

* Recent earthquake in china took a heavy toll

* Recent china earthquake kills million

* China is paralyzed by new earthquake

* Death toll in China exceeds 1000000

* A new powerful disaster in China

* A new deadly catastrophe in China

* 2008 Olympic Games are under the threat

* China's most deadly earthquake

Traffic Control Installation Demo Video

For those who are keen to install the free download of Traffic Control from Mailchannels, but do not know where to begin, I've recorded a quick demo, which I've uploaded to YouTube.



A higher quality QuickTime version is also available for download.

Email Deliverability Tips: Why Can’t My Mail Server Deliver Mail to Yahoo! or AOL?

With constantly growing spam volumes, large email providers have been forced to take measures to reduce spam’s impact on their infrastructure and on their customers. Unprotected email systems are easily crippled by spam outbreaks and it doesn't make sense to overbuild capacity to meet what-if situations.

One way large email providers protect their systems is through the use of a reputation database along with mail architectures which use the database to rate-limit or block emails. AOL, Yahoo!, and others maintain proprietary reputation systems. The following best practices can help maintain your good reputation – improving your mail deliverability not just to Yahoo! and AOL but to everywhere.

If you are having problems with your emails bouncing or being treated as junk, there are three steps you can take which can improve your deliverability:

A) Check the reverse DNS entry for your outbound mail server’s IP address

1. Make sure the forward and reverse match.If your IP resolves to example.com, make sure that one of the addresses that example.com resolves to is your IP.
2. Make sure its set for your server specifically, preferably tothe domain you are sending mail from.
3. Don’t use an address that looks dynamic. A reverse DNS of d192-168-0-1.example.com is considered a much likelier source of spam then “mail.example.com”.

B) Avoid having your server look like a source of spam

1. Old email accounts are often forwarded to a new address. If the address your server forwards to is eventually de-activated, does your email server stop forwarding messages to it? Many won’t automatically. This will mean your server is regularly sending email to an invalid recipient – something that spammers do.
2. If possible, forward email after spam filtering. This can help reduce the impression that it’s your server that originated the spam. It may also help keep your mail queue from filling up with messages which aren’t accepted by the server you are forwarding to.
3. Ensure good list hygiene practices. Whether its someone in marketing sending email from their desktop or a properly managed list using list management software, your responsibilities are the same. Only email people who actually want to hear what it is you are saying to them. Promptly handle and always respect any requests to unsubscribe. Unsubscribe addresses which bounce repeatedly. Comply with CAN-SPAM.

As an email server administrator, configuring your server correctly can go a long way to following these guidelines, but that is only one piece of the puzzle. Your users needs to be educated, and your mail server’s logs need to be reviewed periodically. Your pro-active approach will help avoid having your user’s productivity impacted when they can’t email their contacts.

MailChannels Offers Load Testing Tool

Many of you by now have heard, privately at least, that MailChannels has developed an email system load testing service. We developed this service initially for ourselves (to test the scalability of Traffic Control), but have recently started offering it to existing and potential customers.

The load testing tool works by spawning up to 100 server instances with Amazon's Electric Compute Cloud (EC2), each instance of which spawns hundreds or thousands or SMTP clients which throw millions upon millions of SMTP connections at a target mail server. The clients can be configured to behave in a number of ways, which allows us to simulate real-world email traffic patterns.

We can generate truly earth-shattering email system loads with this service.

For example, when we threw our traffic at a Postfix/SpamAssassin installation (delivering to /dev/null) running on an eight-core box with 15GB of RAM, Postfix quickly queued up about an hour's worth of email. The load average on the machine was 117.5, with 6GB of RAM being eaten up by the Postfix and SpamAssassin processes.

To find out how your mail server scales, please fill in our inquiry form to set up a load test.

New MailChannels Web Site

Yesterday afternoon, we launched a refreshed mailchannels.com website, which we have been working on tirelessly for several months. The new site provides a great deal of new content, designed to explain how our email traffic shaping technology works, and to whom it should be of interest.

As with most web site launches, there were some glitches with ours - the most notable being that our download link (mailchannels.com/download) was returning a 404. With a few last minute mod_rewrite hacks (thank you again, Apache Software Foundation), the download and other links are functioning, and we look forward to any feedback you have on our new content and site layout.

Thanks,
Ken

Spamhaus uses court filing to connect the dots on an "anonymous" e360 netblock

Spamhaus, the London-based operator of the Spamhaus block list (SBL), today released an incident report concerning their listing of 71.5.99.0/25 (SBL52363) as well as a related block ( see SBL51828). These two Spamhaus listings are significant because the two blocks were anonymous until their owner, e360 Insight, used the courts to request that Spamhaus unblock them.

e360 Insight obtained a judgement from an Illinois judge in September, 2006 forcing Spamhaus to remove from Spamhaus' block lists, at e360's request, any IP blocks owned by e360. It's ironic that e360 requested the removal of the blocks described in SBL52363 and SBL51828 because these blocks have been shown by Spamhaus to be sources of spam, in contravention of the CAN-SPAM Act of 2003. In other words, the spammers have tried to use the courts to their advantage, but have instead succeeded in having their own spamming activities validated as such.

From the incident report:

Nothing in the IP address registration, domain registrations, nor even in the spam sample itself, even remotely showed any connection with e360 Insight or David
Linhardt. At the time of creating the record it was technically impossible for
Spamhaus to know that the IP range the spam was coming from and these domains were owned by David Linhardt.

Only when the Plaintiff's counsel sent Spamhaus' counsel a demand for SBL record
SBL52363 to be removed, claiming it was in violation of Judge Kocoras' order to
never list David Linhardt's domains or IPs, did Spamhaus then know that SBL52363
and therefore the 80 anonymous domains and IP range belonged to David Linhardt.

In effect, e360 Insight has held up their hand and acknowledged through the courts that they are spammers! For more background on e360 Insight, read the Spamhaus entry at Wikipedia.