Is Postfix Anvil(8) as good as it gets for spam DDoS protection?

Maybe you've had to deal with a client that hammers your server with simultaneous sessions, or excessive requests in a given time period. In this post, learn how Traffic Control compares with Postfix Anvil.

What is Anvil?

Anvil is a light-weight connection rate control mechanism for Postfix, which allows you set limits on how hard someone can hammer your Postfix server. Anvil was created in response to the dramatic up-tick in spam that started happening several years ago, as a way to protect Postfix installations from high connection concurrency and resource exhaustion. It is commonly recommended as the preferred solution to loading issues relating to spam. Anvil is essentially a side-car process that communicates with Postfix to maintain various counters relating to the hosts connected to your Postfix server. Postfix queries anvil to update and retrieve these counters, and take appropriate action to limit the damage from abusive hosts.

What is Traffic Control?

Traffic Control is a highly scalable SMTP proxy server that seamlessly integrates with Postfix. It increases connection capacity so that thousands of concurrent connections can be efficiently prioritized with negligible system load. Legitimate connections are processed right away, known spam is rejected, and suspicious connections are slowed down causing the vast majority of spammers to give up before completing message delivery.

Traffic Control integrates seamlessly with Postfix using the XCLIENT command (see www.postfix.org/XCLIENT_README.html), front-ending SMTP connections, and applying TCP traffic shaping to suspicious connections before passing on legitimate email to Postfix. Traffic Control is implemented using a very efficient libevent-based asynchronous IO layer, which enables handling up to 25,000 concurrent SMTP sessions with low overhead.

Comparison

Feature	Postfix Anvil	Traffic Control
Description	Works in conjunction with Postfix to maintain and enforce connection, message, and other rate limits on a per-host basis.	Applies TCP traffic shaping and connection multiplexing, increasing the capacity of Postfix to handle 100,000 or more concurrent connections, while eliminating botnet connections with 99.6% effectiveness.
Method of Operation	Receives connection statistics from Postfix, which are maintained in a database and reported back to Postfix via a TCP socket. Postfix enforces rate limits based on the counts reported by Anvil.	Receives SMTP connections, assessing their reputation and behavior through a commercially supported reputation network and set of customizable triggers. Contacts Postfix via SMTP to validate recipients and other SMTP commands in real time, and finally delivers messages to Postfix if the sender adheres to the SMTP protocol and persists long enough to get its message delivered.
Effectiveness against botnets	Rate limiting effectively stops high volume senders from abusing Postfix. Protection against botnet-based attacks is minimal, because individual zombies typically "fly under the radar," making only a limited number of connections.	Hits botnets where it hurts, tying up essential SMTP connection resources and causing 99.6% of zombie-based connections to abort before message delivery has taken place. Abusive high volume senders are forced to wait up to 10 minutes for message delivery, greatly reducing the impact of their traffic on Postfix and downstream users.

Of course, Traffic Control isn't open source, but that doesn't mean it's not easy to get a hold of. If you run a large mail installation and want to try Traffic Control, just fill in our straightforward demo request form and we'll hook you up with an evaluation copy.

Survey Results from Parallels Summit 2009

For those who attended Parallels Summit 2009 in Las Vegas way back in February, you probably witnessed our shameless giveaway of NERF guns in exchange for survey responses. Here’s some interesting statistics to share.

77 surveys were collected in-person at the MailChannels booth. 88% specialize in Servers and Software; 40% were executives.

• When asked how respondents defined spam, the majority chose “any email they did not request,” the next popular answers were “email that violates the CAN-SPAM Act,” and “email promoting stocks, pills, etc.”
• 56% use open source software like SpamAssassin to stop spam, followed by commercial appliances (e.g. Barracuda), and then commercial software (e.g. PureMessage).
• The two biggest challenges to hosting email are lowering infrastructure costs and clean inboxes. 36% said they used more than 51 servers to provide mail service. When asked how often spam affected the availability of their email services, 38.1% said a few times a year, 28.6% said monthly, and 19% said weekly.
• 33.3% said their anti-spam system was a work in progress, while 42.9% said they were looking at new solutions.

The top three solutions respondents believe will improve performance of their mail system:
1) Spam pre-filtering
2) More filtering appliances
3) Restricting bandwidth to spammers

• 30.1% of respondents said that they could save $100K or more per year if they could reduce the number of mail servers and filtering appliances by 80%.
• Resources where people learn about tech are Web Hosting Talk (61.1%) and theWHIR (44.4%).

Conducted bi-annually at HostingCon and Parallels Summit, this survey presents information concerning the trends and perceptions of web hosting professionals. For more information about the survey please don't hesitate to contact us.

What timeouts should I use in my mail server to get email through Traffic Control?

As our customer base grows, we are increasingly coming into contact with email marketers who want to know how they should configure their outbound mail servers in such a way that they can still deliver email across a traffic shaped connection. I thought it would be useful to put together a brief post instructing ESPs on the SMTP timeouts, which Traffic Control is designed to respect so that all legitimate senders have a chance to get their email delivered.

RFC 5321 (the successor to RFC 2821 and RFC 821) very clearly defines the timeouts that mail servers should respect. Here's the relevant text from the standard:

4.5.3.2.  Timeouts

  An SMTP client MUST provide a timeout mechanism.  It MUST use per-
  command timeouts rather than somehow trying to time the entire mail
  transaction.  Timeouts SHOULD be easily reconfigurable, preferably
  without recompiling the SMTP code.  To implement this, a timer is set
  for each SMTP command and for each buffer of the data transfer.  The
  latter means that the overall timeout is inherently proportional to
  the size of the message.

  Based on extensive experience with busy mail-relay hosts, the minimum
  per-command timeout values SHOULD be as follows:

4.5.3.2.1.  Initial 220 Message: 5 Minutes

  An SMTP client process needs to distinguish between a failed TCP
  connection and a delay in receiving the initial 220 greeting message.
  Many SMTP servers accept a TCP connection but delay delivery of the
  220 message until their system load permits more mail to be
  processed.

4.5.3.2.2.  MAIL Command: 5 Minutes

4.5.3.2.3.  RCPT Command: 5 Minutes

  A longer timeout is required if processing of mailing lists and
  aliases is not deferred until after the message was accepted.

4.5.3.2.4.  DATA Initiation: 2 Minutes

  This is while awaiting the "354 Start Input" reply to a DATA command.

4.5.3.2.5.  Data Block: 3 Minutes

  This is while awaiting the completion of each TCP SEND call
  transmitting a chunk of data.

4.5.3.2.6.  DATA Termination: 10 Minutes.

  This is while awaiting the "250 OK" reply.  When the receiver gets
  the final period terminating the message data, it typically performs
  processing to deliver the message to a user mailbox.  A spurious
  timeout at this point would be very wasteful and would typically
  result in delivery of multiple copies of the message, since it has
  been successfully sent and the server has accepted responsibility for
  delivery.  See Section 6.1 for additional discussion.

4.5.3.2.7.  Server Timeout: 5 Minutes.

  An SMTP server SHOULD have a timeout of at least 5 minutes while it
  is awaiting the next command from the sender.

The most common problem we see with email marketers is that their mail servers fail to wait at least three minutes for each DATA send operation to complete. Probably the second most common problem is senders who seem to set an arbitrary timeout on the delivery of the entire message.

Whatever you do as an email sender, it's important to respect the RFCs if you want to get your mail delivered reliably - including formerly esoteric parts such as the timeout requirements.

Spammers continue to suffer from "premature disconnection"

Today we compiled a graph showing the effectiveness of traffic shaping against botnet spam traffic. The graph above shows how long it took for different spambots to disconnect when they were trying to deliver mail to one of our large customers. This customer slows down traffic from any host identified on the Spamhaus Zen RBL, so this data represents a fairly pure profile of the traffic shaped behavior of spam sources.

What's this all mean?

Spammers continue to rely on the timely delivery of tens of millions of email messages to make a profit from their spamming activity. If a spam bot (i.e. spam sending software) encounters a mail server that won't take its mail quickly, there's the profit from that receiver is vastly reduced. It makes more sense to spam someone who will at least receive a message quickly, even if that message will later be discarded with 99% certainty.

Identifying botnets by how long they last on a slow connection

For extra credit, we cross-referenced hosts in each of the "disconnect spikes" in the above diagram with the CBL's (Composite Block List) excellent database of spam sources and found that each spike corresponds with particular spambot software. For instance, members of the climbot botnet invariably disconnect at the 100 second mark, whereas rustock hosts only last 21 seconds.

Could these mean that rustock suffers from premature disconnection?

Haiku on NANOG

The network engineers are getting restless. Here is some recent Haiku from the NANOG (North American Network Operators Group) mailing list:

Cogent drops packets.

Angry customers call. Twice.

Admin writes haiku.

Cogent makes a mess

My phone rings and rings

Unfornicate this!

Net admins are bored

Nanog lists run wild

Useless spam blow up phone

.. and we couldn't resist writing one of our own, which was so off topic it prompted a call to end the madness:

McColo shuts down

Spammers enjoy holiday

... and then recommence.

Is this the sort of thing that happens just before the IT industry lays off most of its workers, leaving more time for Haiku writing and latte serving?