Greylisting story
Printed From: LogSat Software
Category: Spam Filter ISP
Forum Name: Spam Filter ISP Support
Forum Description: General support for Spam Filter ISP
URL: https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6330
Printed Date: 22 December 2024 at 10:15am
Topic: Greylisting story
Posted By: atifghaffar
Subject: Greylisting story
Date Posted: 22 December 2007 at 10:05am
|
Hello all, Summary: Please give us greylisting capabilities. Details: We have been in deep s**t since the past 5 days. Our servers were hammered with spam 4x 5x times more than usual and our two SFI servers were not able to handle the load at all. Note that before the SFI servers we have a firewall that blocks discriminatively half of the world countries and then it blocks any hosts/networks that were blacklisted by SFI and still tried to connect 10 times to it. (we get this from the logs) Adding more servers will basically open more doors and the situation would stay the same, or even worse the backend (quarantine database) will also suffer more. Our sysadmin has talked always about greylisting and I have been waiting to try it when SF implements it because we are using all the cool features of SF and I dont want to re-invent them. There are many softwares for doing greylisting and using them means that I will have to re-implement spf checking, rbl checking, keyword checking, lose sfdb, etc. Another possibility is to use greylisting proxy that forward the good requests to spamfilter but then we lose the origionating IP address and most of the checks will not work (rbl, sfdb, spf, etc) We have tried greylisting on our corporate domain which we separated form SF for a day to first get the taste of unfiltered mails (spams). We got 25K (TWENTY FIVE THOUSAND) in our catchall mailbox in a day. After we just put a greylisting proxy in front of it (our corp mailserver was not doing, rbl, spf checks anyway. It was implemented with "SF will do the filtering and will only deliver good mails to this server" in mind). Anyway after the greylist proxy implemented we only recieved around 800 (EIGHT HUNDRED) messages in the catchall mailbox. Not bad! During all these experiments, our ISP customers were still suffering because mails were not delivered (recieved by SF because it was too busy), etc, etc. Solution: We have implemented a couple of firewall/greylisting servers. These 2 servers replace the firewall that was sitting behind the SFI servers. Each SFI server uses one of them as their default gateway. The system runs postfix, knows which domains and mailboxes we accept mails for. We use sqlgrey (a greylisting plugin for postfix written in perl that reads/write connections and whitelists from a mysql or some other database ) for greylisting. The database server is a central server that is also used for mailrouting, maillogging etc, so more than one instance of this application can use it. Basically this server will tell everyone (except a few IP addresses of our choice) to come back in 5 minutes on their first connection. 95% will not come back (zombies, hacked machines, other smapnets, etc). The other 5% when will try again after 5 minutes, the mail will be accepted and forwarded to the mailbox (Yes the SPAM will get through). At the same time the application will add this ip address with the user/domain (tripplet) to a from_awl table. Nothing special so far. SF is not getting any mails at all. Every 5 minutes we have a script that looks in the from_awl table for entries in the last 10 minutes and add a NATing rule to forward that IP address or the subnet to the SFI server. This means that every IP that the greylisting accepted now goes no more to the greylisting server but to the SF server. Results: Before this trick we had between 600 to 1000 connections on the SFI servers at almost all the time. After this trick with 35K ip/networks in the list (known rfc compliant mailserver/persistent spammers) now go directly to the SFI servers and I have not seen the connections go higher than 10 (TEN). YES really I was sure that something is screwed up somewhere but its not. We have mails flowing in again normally. So, please our SF overlords, would you please give us greylisting. Thanks. ------------- best regards Atif |
Replies:
Posted By: jerbo128
Date Posted: 22 December 2007 at 2:42pm
 HA HA HA HA HA HA. Funny!
|
atifghaffar wrote:Posted By: LogSat
Date Posted: 22 December 2007 at 6:40pm
|
Ok Atif... if you ask this way... you're making us spoil the surprise... We're alpha-testing the new SpamFilter ISP v4. The main two features are two new filters. 1 - SpamFilter Distributed Content. SpamFilter will have the ability to detect similar emails, and will create hashes (signatures) for each group of similar emails. The signature will be uploaded to our centralized database, much like our SFDB. If we receive reports for the same hash, but the emails that have this specific hash are originating from different IPs, we will consider this hash to be a spam hash. From then on, any emails with the same hash will be thus rejected (the theory is that legitimate servers will send their newsletters and mailing lists from the same origin IP, not from different networks... Thus if the same email (or similar - as again SpamFilter is able to group together emails that have similar text in them) is being sent from multiple networks, chances are it is not legitimate. 2. GREYLISTING (but as usual it will be a LogSat's "flavor" of greylisting...) If anyone is interested in the beta, please email us with your order number and, most likely starting from tomorrow, we'll be able to provide it to you (with an option to enable the greylisting as it's turned off by default). ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: atifghaffar
Date Posted: 22 December 2007 at 9:13pm
|
Roberto, Sorry for spoiling the surprise ;-) I read from one your posts "Our only option would be to implement greylisting, but only for single-server configurations" Is this still the case, or did you find a way to sync it between all servers. (sqlgrey simply puts everything in the db and make lookups there). I will send you an email with the order number for the alpha to test the greylisting. For the SFDC: Is is similar to DCC (http://www.rhyolite.com/anti-spam/dcc/) If yes, why re-invent it instead of implementing a dcc client in SF? ------------- best regards Atif |
Posted By: LogSat
Date Posted: 23 December 2007 at 9:31am
|
In this beta, the "GreyListAllowed.txt" file that is used to hold the list of IP addresses that has passed the greylisting stage and are allowed to make connections is self-maintained and handled by the running SpamFilter. It cannot be changed by external application. In the next betas we'll work with SpamFilter Enterprise to have this list stored in the database and distributed amongst the various SpamFilters. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: atifghaffar
Date Posted: 23 December 2007 at 10:08am
|
Roberto then is it ok if this file is on a shared disk and available to all SF instances? ------------- best regards Atif |
Posted By: LogSat
Date Posted: 23 December 2007 at 10:20am
|
Sorry, it's currently not possible. SpamFilter will only write to that file, and will only read it upon startup. SpamFilter will not reimport it while it's running (not in this beta...).
------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: atifghaffar
Date Posted: 23 December 2007 at 11:36am
|
Roberto, as long it is remedied at some timi in future its fine. ------------- best regards Atif |
Posted By: atifghaffar
Date Posted: 24 December 2007 at 10:18am
|
Roberto, I have installed the beta. It is working very well. ------------- best regards Atif |
Posted By: jerbo128
Date Posted: 24 December 2007 at 2:28pm
|
Roberto,
Installed beta over a SFE.730 Installation on our testing server. Running MSSQL 2005. SQL is located over a WAN link.
Upon first start of service, machine cpu goes to 100%. I let it stay there for about 20 minutes. Ram usage on that process was changing. After 20 minutes, I gave up and killed the process. Started the service again and same result.
GUI does not show, however tray icon does show.
SFE process using 35,000 K of Mem with 6 threads
Database stats:
bl_ips - ~200K rows
domains ~100 rows
authorizedUsers - ~3000 rows
bl_domains - ~5000 rows
All domains using same settings
All other tables are small (except for tblmsgs and tbl_quarantine)
Normal startup on SFE.730 would take about 30-60 seconds on this same machine.
now as writing this, GUI did show, but still not accepting connections and CPU still at 100%.
At about 40 minutes, gui did start to show connections and CPU went back to a "normal" usage.
Any idea's?
Jeremy
|
Posted By: atifghaffar
Date Posted: 25 December 2007 at 7:36am
|
Jerbo, For us, it ran without any problems. Stop the service, replace the binaries, start the service. That was it. Mind you we are running SFI not SFE. Please let us know when you or Roberto find out what was causing this. Oh and merry xmas to everyone. ------------- best regards Atif |
Posted By: jerbo128
Date Posted: 25 December 2007 at 3:51pm
 It seems to me that the slowness results when SFE is reloading the larger tables from the database. I did go ahead and install the beta on our primary server (same server that is running the sql) and it behaved the exact same way as the testing server. Took more than 10 minutes to begin accepting connections and show the gui.
I also noticed that each time the tables were automatically reloaded, the CPU on the testing server will go to %100 for 2-5 minutes and then resume back to normal. During this time, it did gontinue to process connections though.
Any thoughts Roberto?
 Merry Christmas Everyone!Jeremy
|
Posted By: ImInAfrica
Date Posted: 26 December 2007 at 5:06am
|
Hi, I've installed SF4 alpha. how/where would i enable the greylisting feature? |
Posted By: ImInAfrica
Date Posted: 26 December 2007 at 5:20am
|
sorry all. Have found it under Settings.. Configuration.. Under 'Global Options - Apply to all domains' second option from the top. |
Posted By: atifghaffar
Date Posted: 26 December 2007 at 12:02pm
|
Roberto, Been running the greylisting since a couple of days. Working fine. Well I have no complains from the customers (They are all on holidays) I see that the fine GreyListAllowed.txt has grown to around 8MB on my primary MX. I know its alpha and we are not supposed to tinker too much with it but hey I have to work duing the holidays. Couple of questions. Is there a setting to put a class in greylist allowed after x amount of hosts have been identified to understand greylisting. For example; I have 116.118.20.121~39441.1153457523 116.118.20.132~39442.1142152546 116.118.20.160~39442.1204926968 So lets say an admin decides that a class C can bypass greylisting if atleast 5 hosts are identified. Will this be in the future versions? PS: What is all the numbers after the ~ thanks. ------------- best regards Atif |
Posted By: LogSat
Date Posted: 26 December 2007 at 4:25pm
|
Jeremy, Do you see this delay only in the beta version, or in the 3.5 versions as well? ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: jerbo128
Date Posted: 26 December 2007 at 5:05pm
|
This is a beta only issue.
It may take up to a minute to load tables on the 3.5, but much much longer on the beta. Once it has loaded, It appears to be fine (until it reloads a table).
Roberto - 2 beta issues:
I am getting a fair number of false positives on the SFDC filter. Out of ~500 in the quarantine, I quickly found 14-15 emails that should have been delivered. Is there anything that I can do that will help you with this? (Actual Emails, logs, etc)
Also, my primary server did quit accepting connections today. It did continue to send quarantine-force-delivered emails and perform tasks (such as cleanups, corpus, etc). I restarted the service and as of 15 mintues - still no GUI. Want logs?
Jeremy
|
Posted By: atifghaffar
Date Posted: 28 December 2007 at 10:57am
|
Story update: Thanks Roberto for the quick access to the beta. We have moved our corporate domain greylist server to SFI beta and its performing fine. We have also moved the 2 SFI server for our isp platform to the beta thus removing the greylist before NAT solution that we implemented temporarily. The greylisting is now being handled by SFI. The two firewalls (one each for SFI) are also removed and replaced with the old firewall which again denies access (based on country and on number of attemtps on SFI after being added to local blacklist cache ) before passing the packet to the SFI. Things are sweeeeet again. ------------- best regards Atif |
Posted By: Desperado
Date Posted: 28 December 2007 at 11:19am
Jeremy,
In case you are not aware, Roberto pushed "The Dial" down a bit on the SFDC. I see MUCH fewer false Positives while still getting a good amount of "Hits". ------------- The Desperado Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
Posted By: jerbo128
Date Posted: 28 December 2007 at 12:23pm
Did not know that. Thanks. I will take another look at the DB and see what's happening now.
Jeremy
|
Posted By: mikek
Date Posted: 08 January 2008 at 5:09am
|
just installed the beta and am very impressed! the performance gain is great, before I usually had 10 - 15 concurrent connections which had to run through the various filters and now I'm down to 1 - 2... great work (as always) Roberto! cheers, Mike ------------- Mike Kellenberger Work: http://www.escapenet.ch - http://www.escapenet.ch Private: http://www.kellyburger.com - http://www.kellyburger.com |
Posted By: dcook
Date Posted: 08 January 2008 at 9:45am
|
I just installed the beta yesterday and our quarantine levels are down 80% per hour. If you are using MYSQL -- here is a way to get a count per hour on qyarantined messages:
select MsgInfo, Count(MsgHour) MsgCount
from (select DATE_FORMAT(MsgDate,'%m-%d-%Y %h %p') as MsgInfo, DATE_FORMAT(MsgDate,'%Y%m%d%H') as MsgHour From tblquarantine) as msgtable Group By msgtable.MsgHour Order By MsgHour Desc  Very impressed thus far. I also sent an email to my hosted clients announcing the greylisting as a positive step in spam reduction and to expect an email to be delayed a few minutes. I've had several atta-boy's from the clients responding to this email.------------- Dwight www.vividmix.com |
Posted By: LogSat
Date Posted: 09 January 2008 at 1:44pm
|
FYI - an updated beta is available in the registered user area.
------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: StevenJohns
Date Posted: 10 January 2008 at 6:12pm
|
>>In this beta, the "GreyListAllowed.txt" file that is used to hold the list of IP addresses that has passed the greylisting stage and are allowed to make connections is self-maintained and handled by the running SpamFilter. It cannot be changed by external application. In the next betas we'll work with SpamFilter Enterprise to have this list stored in the database and distributed amongst the various SpamFilters. I hope that this does not mean that we need to run SF in enterprise mode with the database enabled as we run SF in SFI mode, forwarding emails to an internal server??
By the way, just testing the greylisting....looks good so far.
|
Posted By: Desperado
Date Posted: 11 January 2008 at 12:48pm
|
All,
Just wondering if I am the only one seeing a large (huge really) decrease in ATTEMPTED connections ... almost like the Spammers add my servers to a "don't bother trying again" list. I do know that many applications will add servers to a "suppression" list but usually only after getting a "hard" bounce - 5xx code.
Anyway, after a large amount of log parsing I have set the following settings and am really enjoying the results.
Primary MX server:
GreyListInterval=420
GreyListLimboHold=8 GreyListAllowedHold=30 Secondary MX Server:
GreyListInterval=600
GreyListLimboHold=8 GreyListAllowedHold=2 Still zero complaints from my customers! ------------- The Desperado Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
Posted By: dcook
Date Posted: 11 January 2008 at 1:20pm
|
Hi Dan, Our hits have decreased also while the number of items quarantined while greylisting has been somewhat constant.
We have had only one customer complain and it was due to the mail not being retried more than once. But the customer had the issue with three different ISP's?
------------- Dwight www.vividmix.com |
Posted By: Web123
Date Posted: 11 January 2008 at 1:24pm
|
hi,
No problems here either, spam in quarantine about 80 less now with greylisting active
 /Kim
|
Posted By: LogSat
Date Posted: 11 January 2008 at 5:05pm
Do not worry, with SpamFilter ISP "standard" we will not be requiring the use of the database for anything else other than the quarantine database. All filters will always work without the need of the database. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: Desperado
Date Posted: 12 January 2008 at 2:24pm
|
Roberto et al,
First, I want to state once again, how effective the GreyListing has been for us. Here are some observations after MANY hours of log analysis.
Over the first 6 days, our inbound connection attempts went way up as most messages had to do 2 attempts as expected. Over the 6 days, the connection count went down and became asymptotic as the GreyListAllow list populated ... again, as expected. As of today, our quarantined items has reduced to one quarter, dramatically reducing the load on our SQL server while the actual, delivered good mail quantity remained at it's normal levels. After the first few days I relaxed several of my RegEx filters based on False Positive reports (automatically generated every time a customer pushes a message out of quarantine).
Bottom line is that we have reduced our False Positives to about 0.0095% while our server resources has been reduced to about 1/5 of the level it was running prior . Oh, and customer complaints ... ZERO!
This is just getting better and better. Thanks! ------------- The Desperado Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
Posted By: WebGuyz
Date Posted: 12 January 2008 at 9:50pm
|
Yep, same positive results here after about 6 days. My own email quarantine was averaging about 60-70 new quarantined items a day and now its about 3-4 a day.
So the greylisting picks up the single shot spammers and the blacklist cache cleans up the repeat offenders who got thru the greylist. A good 1-2 punch that spammers will find hard to get around. And with the SFDC cranking up V4 SFE is getting to be an even better product. Thanks for listening and improving it!!
 ------------- http://www.webguyz.net |