How did I get 412,139 spam messages in my Gmail spam box?

I recently noticed that some messages were taking up to 18 hours to reach my Google Apps mailbox. I filed a ticket with Google Apps support, and they began investigating the issue, eventually upgrading it to a “senior technical support team” (this is code for: not a straightforward issue). Email was being delayed between the second to last and last mail server hop, as evidenced by Received headers in each delayed message.

As I was poking around the Gmail interface, trying to figure out whether I had done something stupid to cause the delays, I noticed that there were a whopping 412,139 spam messages in my spam folder. In my opinion, this is just a staggeringly large amount of spam for a single user to receive. I clicked on the spam folder and began paging through the entries. The first few pages looked normal – five false positives, and a bunch of pharmacy stuff. But on page six, I noticed an endless stream of delivery failure notices:

Delivery to the following recipient failed permanently:
     nobody@mailchannels.com
Technical details of permanent failure:

pinging-your-traffic-control-pls-ignore@mailchannels.com has exceeded its quota for sending messages to mailchannels.com.

Ahh – these are messages sent by a testing script that we were running at a customer site to validate the correct configuration of their policy routing tables to enable our transparent antispam system. But why was I getting these delivery failure notices? Turns out that Google Apps sends DSNs for unknown users (and the “nobody” user was until recently an unknown user) to the domain’s super administrator – that’s me. And why were the deliveries failing? Because nobody@mailchannels.com was configured as an empty Google Group – i.e. a group with no recipients.

I think it’s safe to say this gaff was not Google’s fault. And kudos to their systems for handling more than 400,000 spam messages into just one user’s spam folder in the course of a couple of days.