Email & Spam Filtering Stats: Jan 12, 2006

I wrote earlier that I hoped to do some side-by-side looks at how SpamCop, Yahoo & Gmail all handle spam fighting. Here’s the rundown for the mail received yesterday from around 12:30pm UK time to 10:30am today, with the summary being that SpamCop captures the most spam with the least amount of false matches. But that might also be helped by training I’ve done with it over time. Here’s the rundown:

What do the figures on the chart mean? Here we go:

Inbox: How much mail was in my inbox when I checked at each place at the same time today. Gmail is the exception. Since I download email from Gmail throughout the day, my inbox was empty. I’ve estimated how much would be in there if I had NOT downloaded by looking at the total mail I received at Yahoo and SpamCop (about 665 items) and using that as an estimate. Why’s the inbox figure important? It shows how much mail each system forced me to deal with. If a lot of spam gets through to your inbox, more work. SpamCop would have saved me the most.

Spam: Shows how much spam each service filtered. The more spam filtered, the better — EXCEPT if there are a lot of false matches, as explained below. SpamCop did the best job in spam catching. Some additional notes:

• Yahoo uses what it calls SpamGuard, and it’s an off or on affair, no degrees, no way to crank it down to 2 or up to 11. If you pay, you can get what’s promised to be better protection with SpamGuard Plus.
• Gmail has spam filtering on by default. You can’t turn it off or adjust settings.
• SpamCop has a wide range of options. You can use none of them or all. I currently have it to use SpamAssassin with a limit of 5, plus to check against the SpamCop Blacklist, the DSBL open relays and blacklists for South Korea, China, Nigeria, Argentina, Brazil & the SORBS list. This degree of control is nice, but the options and why you might use them aren’t really explained.

False Match: Shows how many items were considered spam and held when in reality, they were legit mail. Why’s the false match rate important? The higher the false match rate, the more work is required to manually check and make sure important messages weren’t missing. More for each service:

• Yahoo in particular savaged newsletters I receive. Search Engine Guide, iMedia, MediaPost, MarketWatch, my weekly Odeon cinema listings and our own SearchDay newsletter all got nabbed as spam. Four emails from people responding to messages I’d sent were also held. There is a way to flag things held as “Not Spam.” I believe that over time, this would help train Yahoo not to reject such material. I may test this in the future.
• Gmail had no false matches. It also caught more actual spam than Yahoo, so points for that. However, it didn’t catch as much as SpamCop. Like Yahoo, Gmail has a “Not Spam” button that I suspect if used may whitelist things and prevent them from being caught in the future.
• SpamCop grabbed the most spam with the lowest false match rate, so a pretty good compromise. Like Yahoo, it nabbed the Search Engine Guide newsletter. It also grabbed two notifications from my Yahoo Groups mailing list, a message to that list and one response to a message I’d sent someone else.One thing I love about SpamCop is that junk mail is sorted alphabetically by default. It makes it very easy to see all the non-English spam I’m getting, plus see the duplicate messages and so on. I could easily do the same at Yahoo (in the beta I used) by clicking to sort by subject. Gmail has no ability like this.SpamCop also has the ability to “Release & Whitelist” items similar to Yahoo & Google. I’ve used this over time at SpamCop, so that is one reason why it might have a lower false match rate than Yahoo. I currently have 210 addresses on my whitelist. I might delete these and start fresh in the future, to better compare. I also have three addresses on my blacklist.

Total Mail: The total amount of mail I received. My “real” mail is uncertain.

Gmail sent me 396 items, and some of those were definitely spam that got through. MailWasher, which I use to prescreen further, has a statistics report window showing what I’ve deleted. Estimated the best I can, I’d say about 100 items of spam got past Gmail. So call it 300 items, which is pretty close to what was in my SpamCop inbox.

Why not take the SpamCop inbox figure as “real” mail given the high amount of spam it pulled? I know from experience some spam still gets through. Using the SpamCop figure, I’d say my real mail was about 200 items, based on a typical day. So overall, 200-300 items, that’s my estimate of what I dealt with yesterday.

% Spam Caught: The higher the better, assuming the false match rate isn’t high. Percentage comes from spam caught divided by total mail received.

% False Match: The lower the better, assuming you also have a high spam caught rate. Percentage comes from false matches divided by total spam caught.