In 2011 I saw a dramatic reduction in my daily spam e-mail volume, just 21,744 messages, down from 63,436 in 2010. Seems as though the takedown of the Rustock botnet was the major reason for the reduction as seen in the graph below. 60 spam messages a day is manageable, especially when I only deal with them once a day when I get the report from Postini. A few false positives and false negatives but compared to years past of 240 spam messages a day I'm much happier.
Another year, another deluge of spam. This is my second year using Postini for spam filtering so while I don't have detailed statistics I can at least compare to last year. For 2010 Postini flagged 63,436 messages as spam or 45% more than last year. My incoming spam rate spiked up to 250 a day in April, May, and June before finishing the year with an average of 170 per day. A quick Google search confirms that some of the biggest dips in the graph below are tied to known spammers being shutdown.
Since I switched to Google for handling my email I don't have as fine grained tracking of spam as I did in years past. My first line of spam defense is Postini which caught 43,683 spam messages for 2009. I get very few false positives from Postini so that number is about spot on. My GMail spam folder currently holds 406 spams messages received in the last 30 days. If I do a rough calculation, I'd say GMail caught another 3,263 spam messages. Postini drops certain spam messages into the bit bucket before I see any trace of it which contributed to only 45% of my email for 2009 being spam. The number of spam messages a day though is marching upwards. Minus a few drops when a spam botnet is shutdown the trend for the year isn't pretty as the graph below shows.
Doing some quick command line hacks I extracted the top 10 spam subjects (ignoring case) that I received in 2009 according to Postini. Each subject accounts for 0.25% of the spam messages I received with no subject accounting for 0.88%. All told just under half of the spam I got used a unique subject.
| Count | Subject |
|---|---|
| 396 | [no subject] |
| 127 | free cialis |
| 124 | age is no longer a barrier for me in bed |
| 123 | pay shipping and get your trials |
| 119 | le_vrai_jeu_se_joue_seulement_lm-`_om-y_il_a_commencm-i:_m-`_las_vegas!_ |
| 115 | we know you want the free cialis |
| 115 | pay shipping for your erotic nights |
| 115 | don't pay anything for your pills for 15 days |
| 114 | feel 10 years younger in bed today |
| 114 | do not underestimate the value of free pills |
Using space as a word delimiter and lowercasing everything, these were the top 10 words found in my spam subjects. Not nearly as interesting as the full subject lines.
| Count | Word |
|---|---|
| 8854 | your |
| 6988 | for |
| 6031 | free |
| 5426 | the |
| 4851 | to |
| 4666 | a |
| 4600 | you |
| 3587 | and |
| 3504 | get |
| 3014 | in |
I woke up this morning to find over 4,000 messages in my inbox. Seems I was on the receiving end of an email bounce bomb. Basically someone was sending out thousands of spam email messages and forged my email address as the sender/return-path for the messages. Given how these things work I'm sure my address was randomly selected out of some list, but my cynical side makes me wonder who I wronged :) At last count I've gotten over 17,000 messages. The incoming message rate is subsiding but at its peak they were coming in at 40-50 a second. Crazy.
Another year gone by, another year of spam filtering. The overall spam percentage for 2007 is almost the same as it was in 2006, but the volume has increased.
In 2007 I received 111,095 emails. Of those 87,623 were spam. Between SpamAssassin and procmail 86,206 (98%) of the spam was automatically filtered while I had to manually flag 1,417 emails (about 4 a day). Overall that means 79% of my incoming email in 2007 was spam, which translates to about 240 spam messages a day. Compared to 2006 that is a 16% increase in total incoming email and an 18% increase in spam.
Since I used the same spam filtering techniques for the entire year the graphs look more regular.
I'm at a loss about the dramatic drop off in spam around June.
The trend at the end of the year doesn't look good if it isn't just a seasonal spike.
The peak and valley pattern is due to day of week fluctuations. If I look at the amount of spam I got for the year based on the day of the week each gets about 12,000. The amount of legitimate email varies such that the weekday totals are about 4,000 more than the weekend resulting in the higher percentage of my email being spam.
For 2006 I kept very detailed statistics on spam. Below is a summary of some quick analysis that I did. I have a bunch of other data so if there is something else that you are interested in let me know and I'll see if I can extract that information.
First the high level numbers: 95,706 total emails for 2006 of which 74,458 were spam or roughly 78%. I should consider myself "lucky" since the average usually reported is around 90-95%. This translates to about 204 spam messages a day.
I whipped up some GD images to show these spam numbers in action. The spike in March is due to a high volume mailing list that I temporarily signed up for.
Prior to the middle of August of this year my spam filtering setup was pretty bad. As a result up 60 spam messages a day were getting through and had to be manually flagged. After updating my spam filters, I'm now only manually flagging 5 messages a day. I don't keep statistics on the number of false positives since I don't want to or care to look at 200 plus messages a day.
This is a day by day breakdown of spam percentage and message counts. Notice the general trend that while the percentage is constant the total number of spam messages per day is increasing.
Over the past couple of days my blog has been heavily hit with comment spam. In the last 10 hours I've gotten more than 100 spam message attempts posted to my blog. I moderate all comments so none of those messages saw the light of day but I still have to deal with them. I've never really tuned the comment spam features of MT and I still haven't. Instead I installed a CAPTCHA system. Previously I had required TypeKey in order to leave a comment but that felt a little to draconian. Not everyone has a TypeKey account or wants to create one. Email whitelists are another option but that still requires that I approve or junk email addresses when they are entered.
I'm all about not even having to look at the spam. While this may mean I automatically trash some important comment or email, I think most people are coming to realize that when 90% or more of all email is spam, people will take drastic measures and at some point a message will be lost. As such I'm now running the SCode plugin. The usability of CAPTCHAs is a concern but unless I go back to TypeKey only comments I don't have another good solution at this point. In fact the use of TypeKey on my system just means that your comment will automatically be posted but you still need to enter the security code in order to post.
Getting junk mail? Want to stop it and save environment at the same time? greendimes is a new company that takes care of contacting junk mailers on your behalf to remove you from their mailing lists.
I finally got around to upgrading the version of SpamAssassin that I use. I was previously running 3.0.2 and am now running 3.1.4 I also spent a bunch of time installing a optional Perl modules to enable additional features of SpamAssassin. Boy am I glad that I did. Prior to upgrading I'd say at least 20 spams were getting through a day. In the last 24 hours I've only see one spam that didn't get flagged and it was as close as you can get to not being considered spam.
This makes me happy. I've never been that good with email to begin with but having to spend most of my email time filtering through my inbox only made matters worse. The few times that I've checked email since upgrading I've almost not known what to do. It's sad, but spam in my email has become so common I've almost forgot what it's like. This is a good problem to have to readjust to.
I do have to say that the documentation that I've come across so far for SpamAssassin has been pretty disorganized. I find this really odd since it seems to be under the Apache Software Foundation, which having used many of their other projects, I've usually been happy with the documentation. I suspect that the SpamAssassin wiki holds the knowledge I'm seeking but damned if I can find it. In any case, less spam is good and AFAIK no false-positives.