The config tab is MUCH better!

Kevin

The image filter caught 3 messages so far today. Two were obvious spam the 3rd was a sales rep blasting out emails to his customers. kida grey area but i'm personally fine with it sitting in there.

Good Job!

Great work on the new 3.0 version.

Will run it a couple of days to find bugs.

found one allready :)

- The shrunk qdb lines are back im sorry to inform you about (you know, the qdb lines reducing height to 2 pixels after a refresh)

R,

Within the AllowedDomainFilterMatrix.txt file I noticed that there are 2 additional parameters at the end with this version.  Can you let me know what they are?

Thanks,
Dan B

Dan B : The first is allow or disable image filtering, second is for SFDB

Roberto thank you very much for the SFDB. If you can make this work I believe this will be a unique feature that sets Spamfilter ISP from any other product on the market.

After reading your description of how this works I thought maybe I would make another suggestion (sorry) :) What about a ranking system. The idea is that if an IP reach X number of hits it stays in the cache longer or maybe for good.

My thought is if an IP is only seen a few times over some period of time it gets flushed but if an IP continues to show up OR reaches a high reporting level it never gets purged and is considered blacklisted. Who knows may be it can get added to the my local blacklist listing.

Lee

allthough i agree with Lee on the suggestion in general, i must point out that lots of people own an infected computer and are completely unaware of this.

Putting an automatic permanent ipblock on would potentially disable a lot of people's mail traffic, as well as the spam/viruses.

Some simply don't have the money to buy decent virusscanners, some don't give a rat's beehind, some (elderly) are unaware of the problem. It just sounds a bit too radical to me, especially when other SF systems are copying the block.

Marco my suggestion has more to do with the way the SF collective (my term for the body of Spamfilter nodes) holds on to IP addresses of those systems that are sending out spam.

If you think about it this is not really any different than any of the Blackhole servers. When an IP is found to be a relaying server or a source of spam it gets blocked and added to the world wide listing. Spamfilter ISP would do the same except it sounds like Roberto has this using a cache mode where it gets purged quite frequently.

My suggestion is that once an IP address reaches some threshold it does not get purged but stays on the list. Maybe the way to handle this is at some point in a future release of 3.0 this could be set by the SF owner.

Say for example I set my threshold at 50 or 100, once an IP is reported by that many nodes in the collective then my Spamfilter server adds that to the static list of blacklisted addresses on my server.

To be honest I am not sure if this is really necessary and we will all have to do testing with this new feature to determine that. My only concern was that cronic spammers don't keep showing up in the list. But in a way this might be self correcting because others will be reporting those addresses.

Either way I am excited to see how this feature evolves and I trust Roberto and the team to make this determination. There will no doubt be changes and tweaks but once this gets worked out I believe this will be another great tool we can all use to keep spam out of our systems.

Lee

hehe, ok, but i do suggest that people should get an opportunity to knock on the SF-hive door (at the risk of beeing assimilated) and ask for a release of the block. but who should act as the SF-queen? :)

Issueing permanent blocks demands someone to control those, and since this is growing into a collective who is to play that role?

But as you said, we'll have to see what Roberto comes up with, something tells me he'll find a good way to handle it :)

Sorry about my earlier brain dead post while the answer was right on the screen to enter 0 to disable.

Maybe this comment will invoke some productive thought:

I would only use this feature if the filter that caused the block was included in the central database and any SFDB blocks enforced on my server matched my filter selections.  

For example, I don't want to block using MX filter, but maybe Desperado does.  My program should not care about SFDB blocks that were caused by MX filter in this case.

Make sense?

.. or "Network Reliability" value tells SpamFilter how many *different* users need to have reported the IP address in question ..

Roberto, i suggest adding a percentage option. So one could say '3' for three users or '2%' for a percentage.

Example: if 2% of all SF users think it's spam, treat it as spam.

As the SFDB network grows, the setting of '3' will become more and more aggressive, since the chance that 3 or more people have ultra-radical filters installed  becomes greater. introducing a percentage of users setting counters this.

Roberto,

I was aware of the points in your comments and to me it would be a useless feature to block based on anonymous uncategorized data. 

Each installation's cached IP blocks are the result of unique filter configurations that match the goals and standards of the individual user.

To block messages based on blocks caused by any filter would be too aggressive for any environment where no false positives is the primary goal. Half of the filters in the program are too aggressive for low or no false positive results.  Some people are so sick of spam they are quite happy with blocking email as well as spam.  These differences in philosophy on how "I" will fight spam and the flexibility of the program is what makes it a superior solution. You should not disregard this value when creating a new feature.

If SFDB is not filter specific, it would be useless. Making it fitler specific would be quite simple.  Use a bit value and the additional data would be miniscule.  SpamFilter should query for common blocks that match the bit value of the running configuration. THEN, you really have something that upgrades the program.

Using a "confidence level" probably would not be needed if the data was filter specific.  A "confidence level" as it is used now does not achieve any type of data accuracy and is virtually useless. All it does is create an illogical damper on using non-specific data.  The path the feature is on now makes it an email blocking feature, not a spam blocking feature.

-MJR

pcmatt,

Hmmm ... I kinda agree with you on the "Bit Value" thing.  I, too have been a little worried about getting overly agressive samples from other users.  HOWEVER, even with the "confidence level" of 2 which is lower than recomended,  I have see very good results.  My worry is how much additional overhead would be required to set a bit based on what filter caused the entry and for now anyway, the existing "bulk" values seem real good.  I am waiting until more usere upgrade before I go to jury.

In less than a day I saw the filter with default confidence level block spam and legitimate email equally. 

The opportunity exists to follow this new feature on a path that is contiguous with the rest of the program and makes it a powerful "spam" blocking filter.  I have no use for a filter where I can not explain to the sender nor the recipient the reason an email was rejected or quarantined.

There are other design issues to be considered too. For example, this feature should likely use UDP only to ahchieve the best performance and lowest overhead.  How are "rogue" users posting garbage to the database going to be detected and blocked?  How will access to the SFDB database be restricted so it is secure?  What other controls and checks will be needed?

Don't misunderstand, I like this idea if it upgrades the program for us all or at least the majority.

I saw lots of false positives from the image filter before I disabled it.  Just like SFDB it was about equal, good blocks and false positives blocking email.

Is there any documentation on what criteria is being used?  How could I tell if an email would be blocked by this filter?

just brainstorming here; how about this idea:

Suppose some users are catching an IP through the keyword filter, while others are catching the same ip through other means.

If the bitvalues Matt suggested AND the keywords are sent to SFDB, which in turn would 'validate' the triggering keywords,once enough users are confirming that the IP is indeed a spammer, and would return the keyword (RegEx), plus the found ip to all SF users. My bet is that loads of IP's would show up and in turn be blocked if confirmed by enough sf users.

This way we would be creating a network that uses all of our combined knowledge and distribute it to all of us.

I do see trouble with this system as well though.. a keyword that is too simple would catch nearly anything, so those will have to be discarded somehow.

what do you all think?

I think we have a great group of thinkers  here.  If the rest of the admins in the world were as vigilant, we wouldn't have a spam problem.  I like the Idea of a bit value, but I am more concerned about overhead, than false positives.  That said, it becomes a matter of choice really.  Marco's percent Idea would help to temper the more aggressive admins as well as offset the less aggressive admins. 

Matt,  as far as telling an offender why they were rejected, I agree that this is an important aspect of spam fighting.  For this reason I would side with the bitvalue approach.  I have given this some thought and aside from the fact that the SPF database would need new fields to id why something was rejected, there is little extra overhead for the clients.  I for one believe that I would not want someone else's list of keyword determining what is spam and what isn't. Also, there are MAPS servers out there that are way to aggressive for my liking.

Thats my 2 cents for now

John,

pcmatt,

I am still finding it hard to understand why you are getting so many false positives as I have not yet seen any with one exception which I whitelisted.  Could the reason be that I have a semi large white list for many valid listservers?  Or, are you including adult content as False Positives.  Strickly speaking, Adult content is not automatically SPAM.  As an ISP, we have many users the WANT their adult content.

Any examples would help.

Obviously we run completely different operations. Luckily Spamfilter is flexible enough to serve many different types of operations. Our users are strictly corporate users and our strategy for fighting spam begain development years before SpamFilter was out.  Our configuration is obviously much different than yours. That does not make yours nor ours invalid nor problematic, just different.

Therefore my comments on hoping for new features that can serve us ALL and not just you OR me, for example.

Dan,

Sorry you took my post the wrong way. I do appreciate your desire to help as you spend your valuable time helping many users on this forum.  I wish I had the time to share detailed information with you on our setup and results.  That would likely serve us both. 

Unfortunately I really don't have the time to do that.

This is major new features being discussed here and I'm trying my best to participate.

OK, so just to clarify. My SF blocks against softtail matches. If an IP gets placed in my IP blacklist cache for failing SPF on softtail x times, then this IP will be submitted to your SFBD also? Just to be clear.

Res=Error=0 & 21 skipping is what I thought, again just wanted to be clear.

And the last point, nothing crucial, and the new filters are brilliant, but just thought I'd ask as I thought it a little strange.

Keep up the good work!