Hello all,

Summary:
Please give us  greylisting capabilities.

Details:
 
We have been in deep s**t since the past 5 days. Our servers were hammered with spam 4x 5x times more than usual and our two SFI servers were not able to handle the load at all.

Note that before the SFI servers we have a firewall that blocks discriminatively half of the world countries and then it blocks any hosts/networks that were blacklisted by SFI and still tried to connect 10 times to it. (we get this from the logs)

Adding more servers will basically open more doors and the situation would stay the same, or even worse the backend (quarantine database) will also suffer more.

Our sysadmin has talked always about greylisting and I have been waiting to try it when SF implements it because we are using all the cool features of SF and I dont want to re-invent them.

There are many softwares for doing greylisting and using them means that I will have to re-implement spf checking, rbl checking, keyword checking, lose sfdb, etc.

Another possibility is to use greylisting proxy that forward the good requests to spamfilter but then we lose the origionating IP address and most of the checks will not work (rbl, sfdb, spf, etc)

We have tried greylisting on our corporate domain which we separated form SF for a day to first get the taste of unfiltered mails (spams). We got 25K (TWENTY FIVE THOUSAND) in our catchall mailbox in a day.
After we just put a greylisting proxy in front of it (our corp mailserver was not doing, rbl, spf checks anyway. It was implemented with "SF will do the filtering and will only deliver good mails to this server" in mind). Anyway after the greylist proxy implemented we only recieved around 800 (EIGHT HUNDRED) messages in the catchall mailbox. Not bad!


During all these experiments, our ISP customers were still suffering because mails were not delivered (recieved by SF because it was too busy), etc, etc.

Solution:
We have implemented a couple of  firewall/greylisting servers.
These 2 servers replace the firewall that was sitting behind the SFI servers.
Each SFI server uses one of them as their default gateway.

The system runs postfix, knows which domains and mailboxes we accept mails for.
We use sqlgrey (a greylisting plugin for postfix written in perl that reads/write connections and whitelists from a mysql or some other database ) for greylisting.

The database server is a central server that is also used for mailrouting, maillogging etc, so more than one instance of this application can use it.

Basically this server will tell everyone (except a few IP addresses of our choice) to come back in 5 minutes on their first connection. 95% will not come back (zombies, hacked machines, other smapnets, etc). The other 5% when will try again after 5 minutes, the mail will be accepted and forwarded to the mailbox (Yes the SPAM will get through). At the same time the application will add this ip address with the user/domain (tripplet) to a from_awl table.

Nothing special so far. SF is not getting any mails at all.
Every 5 minutes we have a script that looks in the from_awl table for entries in the last 10 minutes and add a NATing rule to forward that IP address or the subnet to the SFI server.

This means that every IP that the greylisting accepted now goes no more to the greylisting server but to the SF server.

Results:
Before this trick we had between 600 to 1000 connections on the SFI servers at almost all the time.
After this trick with 35K ip/networks in the list (known rfc compliant mailserver/persistent spammers) now go directly to the SFI servers and I have not seen the connections go higher than 10 (TEN). YES really I was sure that something is screwed up somewhere but its not. We have mails flowing in again normally.

So, please our SF overlords, would you please  give us greylisting.

Thanks.

Ok Atif... if you ask this way... you're making us spoil the surprise...
We're alpha-testing the new SpamFilter ISP v4.
The main two features are two new filters.

1 - SpamFilter Distributed Content. SpamFilter will have the ability to detect similar emails, and will create hashes (signatures) for each group of similar emails. The signature will be uploaded to our centralized database, much like our SFDB. If we receive reports for the same hash, but the emails that have this specific hash are originating from different IPs, we will consider this hash to be a spam hash. From then on, any emails with the same hash will be thus rejected (the theory is that legitimate servers will send their newsletters and mailing lists from the same origin IP, not from different networks... Thus if the same email (or similar - as again SpamFilter is able to group together emails that have similar text in them) is being sent from multiple networks, chances are it is not legitimate.

2. GREYLISTING (but as usual it will be a LogSat's "flavor" of greylisting...)

If anyone is interested in the beta, please email us with your order number and, most likely starting from tomorrow, we'll be able to provide it to you (with an option to enable the greylisting as it's turned off by default).

In this beta, the "GreyListAllowed.txt" file that is used to hold the list of IP addresses that has passed the greylisting stage and are allowed to make connections is self-maintained and handled by the running SpamFilter. It cannot be changed by external application.

In the next betas we'll work with SpamFilter Enterprise to have this list stored in the database and distributed amongst the various SpamFilters.

Sorry, it's currently not possible. SpamFilter will only write to that file, and will only read it upon startup. SpamFilter will not reimport it while it's running (not in this beta...).

Roberto,

I have installed the beta.
It is working very well.

Jerbo,

For us, it ran without any problems. Stop the service, replace the binaries, start the service. That was it.

Mind you we are running SFI not SFE.

Please let us know when you or Roberto find out what was causing this.

Oh and merry xmas to everyone.

Hi,
I've installed SF4 alpha.

how/where would i enable the greylisting feature?

Roberto,

Been running the greylisting since a couple of days.
Working fine. Well I have no complains from the customers (They are all on holidays)

I see that the fine GreyListAllowed.txt has grown to around 8MB on my primary MX.
I know its alpha and we are not supposed to tinker too much with it but hey I have to work duing the holidays.

Couple of questions.
Is there a setting to put a class in greylist allowed after x amount of hosts have been identified to understand greylisting.

For example; I have
116.118.20.121~39441.1153457523
116.118.20.132~39442.1142152546
116.118.20.160~39442.1204926968

So lets say an admin decides that a class C can bypass greylisting if atleast 5 hosts are identified.

Will this be in the future versions?
PS: What is all the numbers after the ~

thanks.

This is a beta only issue. 

Edited by jerbo128 - 26 December 2007 at 5:57pm

just installed the beta and am very impressed! the performance gain is great, before I usually had 10 - 15 concurrent connections which had to run through the various filters and now I'm down to 1 - 2...

great work (as always) Roberto!

cheers,

Mike

FYI - an updated beta is available in the registered user area.

hi,