Hello, networkengineer.

Let me start off by saying I am no expert on SpamFilter. I am still in the stumbling, bumbling stages of the evaluation. But over the past few weeks, I have learned some valuable lessons. There are some things that you can do to increase the rejection rate of bad e-mails, and minimize the possibility of rejecting good messages.

First, remember that the accuracy of the Bayesian filtering is directly related to the accuracy of ALL the e-mails previously rejected or forwarded. So, it is very important that you develop good static filtering rules such as blacklists and whitelists.

Next, if at all possible, blacklist Korea on the Country page. A couple of years ago I figured out that a HUGE amount of junk mail came to us via 211.x.x.x IP addresses. Research showed that the offending addresses were in Korea. So, I decluded at my gateway router all 211.x.x.x addresses assigned to Korea. I suspect that many of those are just open relay mailservers or open proxy server. In any event, Korea is the only country that I have blacklisted, yet an examination of the quarantine list shows roughly 35% of the mail is rejected as "IP address is from blacklisted country."

One of the things that really helped block junk mail on my system was building an extensive "TO Mail" blacklist. A large portion of the junk mail that I have captured and examined is sent to multiple recipients at the various domain names on our network. Nearly always, one of those recipient addresses is either bogus (never existed) or expired (no longer valid). Build a blacklist of those addresses and you will go a long way toward identifying and blocking junk mail that manages to pass other tests like MAPS and No Reverse DNS. For every e-mail that you accurately move from the "good" list to the "bad" list, you will increase the accuracy of your Bayesian filtering. The warning on this one is: when using this filter, do not use expired addresses that belong to a close-knit group. For example, you host abcxyz.com's company e-mail, with users sue, bill, mary and jim, where it is likely that several of those users' addresses will appear in e-mails. If you blacklist one user, I believe the software will block that e-mail for all users to whom that e-mail is addressed.

Also, build a blacklist of prohibited domains. There are some domain names that regularly appear in junk mail. Obviously, you can't blacklist domains like YAHOO.COM or MSN.COM, but there are some that you will never see in legitimate e-mail. Don't try to make a huge, comprehensive list. Just shoot for a dozen of the worst offenders. Experience will be the best teacher in this task. Here are some that are in my list: fromru.com, 21cn.*, 163.com, ccnt.com.

As for your problem of rejecting good e-mails, be VERY careful with keywords. Only include words which you are absolutely certain will never appear ANYWHERE in a legitimate message. My Keyword blacklist only has about 10 entries. For example, you could include "viagra" as a keyword, but what happens when someone sends a joke about Viagra? A legitimate e-mail is bounced. Worse is that the message is flagged as bad, and the contents are added to the corpus as such.

Even adding a spammer spelling of a word such as V1@GRA can cause problems when messages contain 7-bit encoded data. It is possible that a string of characters will appear somewhere in encoded data. The larger the attachment, the greater the likelihood that a "random" string of charaters will match one of your keywords. The longer the keyword, the lower the probability that the character sequence will appear in the encoded block. For that reason, I would suggest that you never have any keyword less than 6 characters long. You can test this by opening a rejected good message in a text editor so that the full header AND any attachments are displayed fully. Then, do a search of the message for some of the shorter words in your keyword blacklist. You will probably find that the character sequence shows up in an attachment.

I recently added Cialis to my keyword list since the word appeared in so many pharmaceutical e-mails that I didn't care if I rejected a joke in the process. I am usually pretty meticulous about predicting unexpected results. But I started getting reports that customers were not able to get their personnel reports from Administaff staffing agency. An examination of the quarantine list revealed that the messages were being rejected for the keyword "cialis". Turns out that the signature lines all contained something like "Staffing Specialist". >> speCIALISt > speCIALISt > speCIALISt speCIALISt speCIALISt << Oops!

No matter what filtering system you use, the degree of its success is going to be related to how well you analyze the e-mails that were incorrectly handled by the program.

Sorry for the long message. I hope this helps.

Dennis

You can solve the cialis problem you mentioned by preceeding the word with a space in the keyword blacklist.

We usually do not edit postings, but as this user is clearly spamming the forum (4 posts within minutes), we will blacklist the IPs used to access the site.