Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - The ONE missing SF feature
  FAQ FAQ  Forum Search   Register Register  Login Login

The ONE missing SF feature

 Post Reply Post Reply
Author
Lee View Drop Down
Groupie
Groupie


Joined: 04 February 2005
Location: United States
Status: Offline
Points: 50
Post Options Post Options   Thanks (0) Thanks(0)   Quote Lee Quote  Post ReplyReply Direct Link To This Post Topic: The ONE missing SF feature
    Posted: 01 March 2006 at 11:23am

I have been using SF for a long time and the product has just gotten better and better. There are so many great filters and features that it does a terrific job of catching the majority of all spam.

Now I know everyone still has things they would like to see added to their wish list but I can only thing of one area that is not addressed at all by SF.

Submit As Spam
This has been mentioned before but SF really needs some method of submitting spam back to SF. When spam does make it through we really need a way to forward emails to a special email address on the SF server that will parse the header and body and flag this as 100% spam.

There are other things I would like to see added (like an interface for humans to use to create keyword/Regex flags) but while there are always ways to make a product easier to use just about every thing you need can be achieved on way or the other with the existing product.

If SF could learn from the Submitted As Spam feature this would reduce the time and effort required to go in and look at headers and manually add IP's to the black or build a lot of custom keywords or Regex searches.

Don't get me wrong having complete control over adding this information manually is fantastic and the SAS feature is not a cure all. But I see this as a capability that is sorely missing from a great product and would like to see more work in this area.

Lee

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 01 March 2006 at 6:02pm
Lee,

It's not as simple as one may think. Here's the major hurdles we see.

1. For most users, the "outgoing SMTP server" in their email client configuration points to the "real" SMTP server used for the company, not SpamFilter. This will mean that any emails they send to a domain owned by that company (i.e. your domain - ex. SubmitAaSSpam@mydomain.com) will go to your SMTP server, not SpamFilter.
To have SpamFilter be able to actually receive emails from your own customers, administrators would have to setup a new email domain (ex. SAS.mydomain.com) in DNS used solely for this reporting purpose, and add this domain to SpamFilter. This is doable of course, but is not something that is "plug and play".

2. In order for the Bayesian filter to learn than an email that passed thru is actually spam, the user would have to submit the *original* email, as-is, with the original headers and body intact. Most of the times this will be impossible, as email clients, especially Microsoft Outlook, completely change the email's source. Submitting a modified email back to the Bayesian filter for learning is a very, very bad thing to do, as this will create incorrect entries in the statistical database, rendering it much less reliable.

3. The only header that is reliable within an email is the one SpamFilter itself adds, reporting the remote IP that sent the email. So the only realistic useful feature that can be obtained by sending the email back is knowing that the IP used to send spam is "bad". Please note that many times, by the time a user actually reads an email and reports it back, that IP will very likely already have been added to the IP MAPS RBL blacklists SpamFilter uses, so emails sent after that time will already likely be blocked.

Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 02 March 2006 at 12:14am

Since we are discussing the one thing missing I was going to see if Spamassassin had ever been considered, or at least a 'hook' so that those adventurous enough could get access to the message stream and put that through their own filtering after SF does its thing.

As Lee said SFI has been getting better and better and the things I used to harp about like greylisting have been replaced with something better in my opinion, namely the blacklist cache.

Keep up the great work Logsat!!

http://www.webguyz.net
Back to Top
mikek View Drop Down
Senior Member
Senior Member
Avatar

Joined: 22 February 2005
Location: Switzerland
Status: Offline
Points: 133
Post Options Post Options   Thanks (0) Thanks(0)   Quote mikek Quote  Post ReplyReply Direct Link To This Post Posted: 02 March 2006 at 2:15am

Roberto,

My 2 cents to your "major hurdles":

1. We are ISPs, so setting up a mail domain for spam reporting is no problem

2. There could be an option to quarantine "good" messages as well and add a unique identifier to the e-mail sent, so that when it is sent back, it can be identified and the original mail can be used to train the bayesian filter

Regards,

Mike

Back to Top
Lee View Drop Down
Groupie
Groupie


Joined: 04 February 2005
Location: United States
Status: Offline
Points: 50
Post Options Post Options   Thanks (0) Thanks(0)   Quote Lee Quote  Post ReplyReply Direct Link To This Post Posted: 02 March 2006 at 9:23am

Roberto I am not familiar with all of the issues involved but it does seem that others are using pieces of this type of technology.

For example Spamcop.org has a spam submission capability and while I am not sure exactly what they are doing some how this info is making it into their database.

I have also used other spam blocking services that provide a Outlook toolbar/icon and it allows you to flag and submit those message back to them as spam.

Quote 1. For most users, the "outgoing SMTP server" in their email client configuration points to the "real" SMTP server used for the company, not SpamFilter.

Well I guess there are number of way to handle this and not sure what makes sense but a simple alias on the mail server could forward spam to spam@192.168.1.100 which would be the address for SF. I would suggest that SF ONLY respond to this email address (defined by the SF admin) to add new spam.

Also a field for Inbound IP address could be added so SF would only accept spam to the above email address and only From the IP defined for the internal mail server.

Quote This is doable of course, but is not something that is "plug and play".

I think this is a very important point. In my humble opinion SF is fantastic but it is far from plug and play. As soon as you have to start creating Regex strings things can get ugly real quick.  I could create an entire thread on this topic alone but let me try to point out a few issues that I believe is important to whether SF continues to be a stand-alone "point" solution or a standard for ISP's.

Strength in Numbers
If you look at your own stats for your Spamfilter server you see that the majority of spam is blocked by MAPS servers.

64937  IP found in MAPS search
16230  IP address is from a blacklisted country
15726  SPF Sender Policy Framework match

So what does this tell us. It tells me that NETWORKING is the key to building an efficent plug and play spam filtering system. If I have to add every rule and and every IP to my own system then SF is not a less useful product.

If we didn't have MAPs and SURBL services then the amount of work involved would be more than anyone person could handle. So my point is SF has to become more automated, more connected and more "plug and play" to use your words.

I believe there are two (and more) ways this needs to happen.

1. The ablility to Teach SF what is spam.
Right now SF can slice and dice an inbound email more ways than Ron Popels vego-matic. Yet on a daily basis I have to go in and add IP's, keywords, honeypot addresses etc. At some point and ISP has to throw up their hands and start looking at Spam based service who will do all this work for them.

Spamfilter is a mature enough product now that it needs to start moving towards ease of use not more bells and whistles. Human intervention is very important in the sense that if I say its spam then my judgement is always better than any filter that SF tries to use. So SF should take my word for it and some how some way parse my submissions and at least make an attempt to use the IP, keywords, some thing to add it to the spam list.

2. Better Cooperation
I have brought this topic up before and its worth raising it again. The stats speak for themselves. MAPS servers work because they learn from each other. Logsat has a tremendous resource at its disposal but you are not taking advantage of it. And this is every copy of SF is a node, it learns and grows every day. But what is missing is the ability for MY SF server to learn from the experience of other SF nodes on the net.

Every Email server and Firewall now supports MAPS and other spam blocking resources. So if SF is going to be unique it has to offer some thing more than great filters.

If Logsat wants to build a killer app and to be able to compete with companies like Symantec and Trend which are a 1000 times your size then you have to leverage your strengths. And your strength is us, your loyal users who have stuck with you. Every user who downloads the SF Trial is a potental node and strengthens your position in the market place BUT only if that node is tied to other SF users. If SF were to focus on allowing SF servers to learn from each other this would put you in league of your own and beyond any other product on the market.

Roberto I don't have all the answers but I will say my comments come from 20 years of marketing in the computer industry. I have competed against companies like IBM, DEC and Microsoft and I tell you from experience the only way to not get crushed is to leverage your strengths. Playing the game on their turf is certain death.

I have a number of other suggestions I would make but they are better provided via phone. I want you to know that my suggestions are sincere and I believe in what you are doing. I admire your passion for your product and your users. My hope is that SF is even more successful and my comments are meant to encourage and help you in this effort.

Lee

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 02 March 2006 at 8:47pm
Lee,

You have some very valid points. It's a long post, which made us think and ponder for a while... Let me "digest" it, and, as usual, let's see what comes out of it... I'll reply in a day or two after we brainstorm it here internally for a while.

Thanks!
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Lee View Drop Down
Groupie
Groupie


Joined: 04 February 2005
Location: United States
Status: Offline
Points: 50
Post Options Post Options   Thanks (0) Thanks(0)   Quote Lee Quote  Post ReplyReply Direct Link To This Post Posted: 03 March 2006 at 12:13am

Yea sorry for the long post. When I get on a roll it just comes pouring out.

But I am pleased that you found some interest in my input and that you took my suggestions in the spirit they were intended.

Lee

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 07 March 2006 at 4:02pm
Hi Lee,

We'll continue to keep the submittal of spam emails that made it thru the filter on hold. The only positive outcome we'd see for the above, would be the blacklisting of recent IPs that were used to submit spam. However usually spammer IPs are very volatile, change a lot, and if they do not change, it's usually a matter of minutes/hours before they are blacklisted by RBL servers anyways.

For your "networking" comments however, you bring very valid observations. We've always tried to stay away from providing a centralized database that SpamFilter would use, but your comments may have changed that.

We're looking to have SpamFilter "upload" its list of the new blacklisted IP cache to our servers. A new filter would then be developed to query the database. We'll need to have "confidence levels" in the filter and in the database, so that this new filter will only block spam if originating from IPs that were reported by multiple users. Encryption will need to be considered to avoid "poisoning" of the database. The uploading of the data would be optional for privacy reasons.

I do not have any timeframes for this, but we'll try to have it ready before SpamFilter v3.0 which is to be released within the next 3-4 months.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 07 March 2006 at 8:03pm

Hey how about supporting vbscript as well as regex as a filter. We are in a windows environment and cscript.exe is native to windows. Then we can have a trully powerful tool! What does everyone think about that?

http://www.webguyz.net
Back to Top
Marco View Drop Down
Senior Member
Senior Member
Avatar

Joined: 07 June 2005
Location: Netherlands
Status: Offline
Points: 137
Post Options Post Options   Thanks (0) Thanks(0)   Quote Marco Quote  Post ReplyReply Direct Link To This Post Posted: 09 March 2006 at 9:57am

One thing i really would like to see implemented is a checksum feature that verifies each incoming  mail's checksum to a 'bad checksum database'

Let me explain a bit more;

Once a mail has been positively identified as beeing a spammail, a checksum number is created and stored in a database. Every 'unknown' mail that spf receives afterwards will also have a checksum generated, and compared to the ones in the database.

If  a match is found, the mail will be rejected as usual, possible added to a blacklist.

Using the first 1 kb of data received to generate and compare a checksum would immediately recognise identical spam from whichever sender.

Anyone who is capable of getting himself made president, should on no account be allowed to do the job. D.Adams
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.160 seconds.