Print Page | Close Window

The software doesn't work quite well

Printed From: LogSat Software
Category: Spam Filter ISP
Forum Name: Spam Filter ISP Support
Forum Description: General support for Spam Filter ISP
URL: https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=4142
Printed Date: 14 March 2025 at 3:59am


Topic: The software doesn't work quite well
Posted By: Guests
Subject: The software doesn't work quite well
Date Posted: 14 August 2004 at 1:53pm

I've been using SpamFilter for quite a long time. So far, my users are still not happy with the result. Either they still have the spam coming in every day or they're not receiving emails that are supposed to come through.

I'm using the free version v2.0.1.302, and using every features I can find in the software to restrict the incoming messages. This includes a long lists of the Keywords and MAPS and attachments. I'm not using much the From, To, IPs, Country  Filters in BlackLists because I don't know where the spam would be coming from.

Even though I'm having a long listsof keywords and MAPS, some of the users especially those who have had their email addresses for a long time still averagely receive 5 spam messages per day.

I've even put the ".info", ".biz" into my keywords list, because most of drug messages would have a link to such domain names. However, there are some out there having links to ".com" and I cannot put ".com" into the keywords list as that will render to too many false positives.

I pretty much have no idea whatelse I should do with this software to make my users happy. I told them that none of the software can block 100% spams, but they told me their Hotmail account never receives 5 spam messages per day which is TRUE! I'm just wondering now what Hotmail has done to gain such a success?

Should I add another spam filtering system on top of the SpamFilter? or I'm not setting it up good enough?

Please advise if you have any ideas. Appreciate!


Replies:
Posted By: Guests
Date Posted: 16 August 2004 at 2:04pm
Hello, networkengineer.

Let me start off by saying I am no expert on SpamFilter. I am still in the stumbling, bumbling stages of the evaluation. But over the past few weeks, I have learned some valuable lessons. There are some things that you can do to increase the rejection rate of bad e-mails, and minimize the possibility of rejecting good messages.

First, remember that the accuracy of the Bayesian filtering is directly related to the accuracy of ALL the e-mails previously rejected or forwarded. So, it is very important that you develop good static filtering rules such as blacklists and whitelists.

Next, if at all possible, blacklist Korea on the Country page. A couple of years ago I figured out that a HUGE amount of junk mail came to us via 211.x.x.x IP addresses. Research showed that the offending addresses were in Korea. So, I decluded at my gateway router all 211.x.x.x addresses assigned to Korea. I suspect that many of those are just open relay mailservers or open proxy server. In any event, Korea is the only country that I have blacklisted, yet an examination of the quarantine list shows roughly 35% of the mail is rejected as "IP address is from blacklisted country."

One of the things that really helped block junk mail on my system was building an extensive "TO Mail" blacklist. A large portion of the junk mail that I have captured and examined is sent to multiple recipients at the various domain names on our network. Nearly always, one of those recipient addresses is either bogus (never existed) or expired (no longer valid). Build a blacklist of those addresses and you will go a long way toward identifying and blocking junk mail that manages to pass other tests like MAPS and No Reverse DNS. For every e-mail that you accurately move from the "good" list to the "bad" list, you will increase the accuracy of your Bayesian filtering. The warning on this one is: when using this filter, do not use expired addresses that belong to a close-knit group. For example, you host abcxyz.com's company e-mail, with users sue, bill, mary and jim, where it is likely that several of those users' addresses will appear in e-mails. If you blacklist one user, I believe the software will block that e-mail for all users to whom that e-mail is addressed.

Also, build a blacklist of prohibited domains. There are some domain names that regularly appear in junk mail. Obviously, you can't blacklist domains like YAHOO.COM or MSN.COM, but there are some that you will never see in legitimate e-mail. Don't try to make a huge, comprehensive list. Just shoot for a dozen of the worst offenders. Experience will be the best teacher in this task. Here are some that are in my list: fromru.com, 21cn.*, 163.com, ccnt.com.

As for your problem of rejecting good e-mails, be VERY careful with keywords. Only include words which you are absolutely certain will never appear ANYWHERE in a legitimate message. My Keyword blacklist only has about 10 entries. For example, you could include "viagra" as a keyword, but what happens when someone sends a joke about Viagra? A legitimate e-mail is bounced. Worse is that the message is flagged as bad, and the contents are added to the corpus as such.

Even adding a spammer spelling of a word such as V1@GRA can cause problems when messages contain 7-bit encoded data. It is possible that a string of characters will appear somewhere in encoded data. The larger the attachment, the greater the likelihood that a "random" string of charaters will match one of your keywords. The longer the keyword, the lower the probability that the character sequence will appear in the encoded block. For that reason, I would suggest that you never have any keyword less than 6 characters long. You can test this by opening a rejected good message in a text editor so that the full header AND any attachments are displayed fully. Then, do a search of the message for some of the shorter words in your keyword blacklist. You will probably find that the character sequence shows up in an attachment.

I recently added Cialis to my keyword list since the word appeared in so many pharmaceutical e-mails that I didn't care if I rejected a joke in the process. I am usually pretty meticulous about predicting unexpected results. But I started getting reports that customers were not able to get their personnel reports from Administaff staffing agency. An examination of the quarantine list revealed that the messages were being rejected for the keyword "cialis". Turns out that the signature lines all contained something like "Staffing Specialist". >> speCIALISt > speCIALISt > speCIALISt speCIALISt speCIALISt << Oops!

No matter what filtering system you use, the degree of its success is going to be related to how well you analyze the e-mails that were incorrectly handled by the program.

Sorry for the long message. I hope this helps.

Dennis

Posted By: Guests
Date Posted: 16 August 2004 at 11:01pm

Hi Dennis,

Thank you so much for the very helpful information which inspired me of new ways to use the software. It is already getting much better after I've done something according to your suggestions. Thanks again!

Danny

Posted By: Guests
Date Posted: 18 August 2004 at 12:13pm
You can solve the cialis problem you mentioned by preceeding the word with a space in the keyword blacklist.

Posted By: johnny5
Date Posted: 20 October 2005 at 3:35pm
You should check into the Aristotle Spam Defense Network.  It identifies servers, using a layer 2 defense network, chokes them down prevents them from sending any spam to your box ever again.  I signed on with the company months ago and have literally been spam and virus free!!  I suggest you check out aristotle's website.

Posted By: LogSat
Date Posted: 20 October 2005 at 5:01pm
We usually do not edit postings, but as this user is clearly spamming the forum (4 posts within minutes), we will blacklist the IPs used to access the site.



-------------
Roberto Franceschetti

http://www.logsat.com" rel="nofollow - LogSat Software

http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP

Posted By: Alan
Date Posted: 21 October 2005 at 3:20pm
SpamFilter is like a Harley motorcycle.  You need to know how to get your knuckles dirty to get the best possible performance out of it.  It just takes a little work to get yourself into how it needs to work for you.  But once you gain experience, you are able to address all sorts of specific issues to your users instead of relying on the judgement of a third party.  Thre are a lot of experienced users here so if you have specific issues, you will find that there is a lot of help available here.  You can also use the search feature to find answers in the forum.
Good luck.


Print Page | Close Window