gmail greylisting
Printed From: LogSat Software
Category: Spam Filter ISP
Forum Name: Spam Filter ISP Support
Forum Description: General support for Spam Filter ISP
URL: https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6731
Printed Date: 27 December 2024 at 3:44am
Topic: gmail greylisting
Posted By: dave.m
Subject: gmail greylisting
Date Posted: 05 August 2009 at 1:44pm
|
We recently downloaded SpamFilterISP and are currently doing tests with it. We are a website design company that hosts sites and e-mails for several companies and would like to migrate off of our existing spam solution that uses sendmail/spamassassin/etc on a Linux platform. Installation went fine, and some tests have come through successfully. We're having an issue with gmail clients being able to send mail to us. I looked through the forums, and saw a couple of other posts related to this, and it appears to have to do with greylisting. If I understand correctly, when a mail comes in from gmail if the IP isn't on the greylisting "whitelist" it's told to check back in X minutes determined the greylisting setting. Apparently, gmail does try back at whatever time interval the gmail servers are set to try back at - but is most likely coming from a different IP address. Eventually this mail gets bounced back enough that it doesn't come through and the sender receives an undeliverable e-mail. We would like to know if anyone has found a way to to whitelist all gmail sending IP's. I know it can't be done by ranges from a previous post, but is there a way to import these into the database perhaps? Or has anyone else found a solution to this issue? Thanks, Dave |
Replies:
Posted By: LogSat
Date Posted: 05 August 2009 at 3:59pm
|
Dave, If you have a list of gmail's IPs, you can prevent specific IPs from being blocked by the GreyList filter as follows. SpamFilter stores the IPs that have been allowed to pass the greylist filter in the file: \SpamFilter\Domains\GreyListAllowed.txt The entries in that file are in the form: aaa.bbb.ccc.ddd~nnnnn.ddddddddddd where aaa.bbb.ccc.ddd is the IP address, and nnnnn.ddddddddddd is a decimal date value. For example: 80.69.70.3~39495.5492583796 The decimal date value indicates the number of days that have passed since 12/30/1899. The fractional part of the value is fraction of a 24 hour day that has elapsed. SpamFilter will delete daily IPs whose date indicated above older than the number of days indicated in the following SpamFilter parameter (60 days by default): GreyListAllowedHold=60 You can manually insert the IP you want in the GreyListAllowed.txt file, and assign it a date that is a few years in the future. For example, adding 10 years would require adding 365x10=3650 to the value indicated: 80.69.70.3~39495.5492583796 plus 3650= 80.69.70.3~43145.5492583796 So, in short, simply add the above line to the GreyListAllowed.txt file. Please note that SpamFilter will only load this file during startup, it does not support changes made to it while it is running. You thus need to stop SpamFilter before making any manual changes to it. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: dave.m
Date Posted: 06 August 2009 at 10:40am
|
Thanks for the quick response. So I guess our options are to add the 65,000 IP addresses that gmail supposedly sends from: 216.239.32.0/19
|
Posted By: LogSat
Date Posted: 07 August 2009 at 12:53am
|
Correct - the greylist filter does not support subnets/wildcards. The reason for this is that the greylist filter works on a TCP level by blocking IPs directly and immediately upon connections. It must be very fast to avoid DDOS attacks, and the most efficient way we could implement it was by having a plain text list in RAM to perform fast lookups. Mixing the list with subnets would cause the lookups to be much slower. Of course adding 65,000 IPs will also make it slower... but not as much as having admins enter 100s of subnets mixed with thousands of IPs. There are however very, very good admins on this forum, so if anyone has any opinions on this we always listen! ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: WebGuyz
Date Posted: 10 August 2009 at 12:22am
|
Roberto,
Understand you want peak performance, but the reality is that we need to be able to compensate for these large farms of smtp gateways, as well as handling greylisting sharing amongst multiple SFE's. I had to setup a non-greylisting version of SFE for a number of customer because of horrible delivery times because of the issue of greylisting and multiple IP's from a single large ISP. That really hurts when customers threaten to leave because of slow delivery when I bust my butt to deliver the best service possible. I would gladly give up performance to be able to not have this happen. What good is performance when your service is called poor because it can take an hour or more for an email to get from point A to B. Customer could care less that I have a high performance spam engine, all he knows is that his business contact who was on the phone with him sends him an email, but my customer does not get it until over an hour later, he calls to tell me my service sucks.  ------------- http://www.webguyz.net |
Posted By: Wayne
Date Posted: 10 August 2009 at 7:18am
|
Roberto I totally agree with WebGuyz! We need this option and I'm also willing to sacrifice performance for that. I cannot believe that this whitelist implementation would took about an hour for most customers. Otherwise i cannot explain how the other anti-spam manufacturers have solved this problem and you know, they have.  Regards Wayne ------------- SF4.5.0.1-beta |
Posted By: yapadu
Date Posted: 10 August 2009 at 10:42am
|
I often feel the delay of having to wait. I know every time I sign up for an online service I am going to have to wait an extended period of time... knowing the email will come in 15 minutes or maybe an hour (or sometimes the next day). Greylisting is doing what it is supposed to be doing though, just having it on the server keeps the CPU load levels down quite a bit by keeping the trash connections from even getting to the server. I suspect there may be a suitable comprise, such as allowing individual IP addresses as well as a class C. That reduces the number of entries people have to make and should still keep the complexity of the lookups down. I don't know how SpamFilter works internally, but I have noticed that IP Addresses are all stored in 123.456.789.123 format in the databases. When dealing with IP addresses I find it much quicker to store them as a numeric base 10 value - some binary format is probably much better than that. Roberto is the master, and knows what spamfilter is doing behind the scenes but maybe storing the values in a different format would allow a speed improvement enough to allow some additions of range based lists. Another option might be to maintain two lists in memory for the greylists, one of just IP's and another of class C network ranges etc. You just need to search two lists, and you know the format of the addresses in both so mixing of IP's vs. Class C list searches should not cause big problems. I also vote for some type of shared list among servers. If you have a cluster of spamfilter servers the issue of inbound greylisting gets even worse, causing even longer delays. Imagine a cluster of 4 servers, some external server connects to server 1, gets rejected. The next time it tries it connects to server 2, then a different server from their cluster tries and gets server 3. Could take a very long time before the message gets through. I'm happy with the grey listing at the moment though. |
Posted By: yapadu
Date Posted: 10 August 2009 at 11:04am
|
Again, I'm just thinking out loud here... If Roberto is storing the addresses in memory as strings, there could be quite a bit of improvement if the addresses were converted to base 10. Then you could use some quick search methods, such as the discussion here: http://leepoint.net/notes-java/algorithms/searching/binarysearch.html |
Posted By: LogSat
Date Posted: 10 August 2009 at 5:45pm
WebGuyz, When you mention "performance", am I interpreting correctly in that you'd prefer to loose some performance in exchange for the flexibility of being able to manually add subnets in the greylist? In regards to the sharing of the greylist amongst multiple servers, this is a *much* bigger challenge to address. Just to get the ball possibly rolling, could you let me know how many entries you currently have in the GreyListAllowed.txt file? ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
WebGuyz wrote:Posted By: WebGuyz
Date Posted: 10 August 2009 at 10:33pm
|
Our greylist file has about 235K entries and service about 5k mailboxes. We have 2 SFE's.
<soapbox>
Greylisting is a excellent tool and really has decreased the amount of traffic hitting our servers, but if it causes long delivery delays and causes perception that your service is 'slow' because sometimes it takes over an hour for an email to get delivered (because it was bouncing around between different IP or in our case multiple SFE's), then it becomes a liablilty.
We have an organ donation firm that needed timely emails from all around the country, but even when they whitelisted everyone they would still get some emails over an hour old because of the greylisting. They threatened to leave so I put them on a copy of SFE with no greylisting. Since then I've had to put 6 other companies on that same SFE with no greylisting because of complaints of slow delivery of emails, usually from senders in companies like IBM and other large corps that have a LOT of different IP's used to deliver mail and which they alternate retries.
I hate it when people tell me my service sucks and I can't do anything about it. I love SFE, but greylisting is its achilles heel in a multi SFE environment and where its not easily possible to put complete subnets in for accepting emails from large ISP's.
</soapbox>
------------- http://www.webguyz.net |
Posted By: yapadu
Date Posted: 10 August 2009 at 11:34pm
| My greylist file on the primary server has 200k, and the backup server has 66k. These have all been automatically added, we have not put in any manual entries. |
Posted By: Wayne
Date Posted: 11 August 2009 at 2:25am
|
So it seems I'm a little smaler, I only have 22k entries  Therefrom are 200 manually added.
------------- SF4.5.0.1-beta |
Posted By: LogSat
Date Posted: 13 August 2009 at 12:00am
|
Within the next 24/36 hours we'll have an early beta for SpamFilter (both Standard and Enterprise) that allows the entry of IP wildcards (ex. 192.168.255.* or 192.168.*) in the GreyListAllowed.txt greylist file. You may contact us at support at logsat.com if you wish to test it before it's released on our website (please include your order number in the email so we can verify the current software maintenance). ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: WebGuyz
Date Posted: 13 August 2009 at 12:10am
|
You know what else would be really good, a way to do a soft reset to reload the greylist file without having to do a complete restart. The same way the spamfilter.ini is read in every minute, but it wouldn't have to be that frequently.
I know, I know, we keep asking for the sky, but when you have over 400 domains and a huge autowhitelistdelivery table when using SFE it take almost 5 minutes to load SFE into memory.
Thanks!  ------------- http://www.webguyz.net |
Posted By: Wayne
Date Posted: 13 August 2009 at 2:11am
Great news Roberto!  Thanx!! ------------- SF4.5.0.1-beta |
Posted By: Desperado
Date Posted: 17 August 2009 at 2:31pm
|
Thought .... are we using the "dnsbl white list"? Can that be used to generate entries in the gray-list allow list that expire in a year or so?
Just letting ideas rattle around in my head! ------------- The Desperado Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
Posted By: dave.m
Date Posted: 17 August 2009 at 4:41pm
|
Roberto, We're currently testing the software (see original post :)) Would like to know if we can get a copy of this beta so I can try and get it approved for purchase. Thanks! Dave |
Posted By: LogSat
Date Posted: 17 August 2009 at 10:13pm
|
Dave, Yes, it's not going to be a problem. If you can you please contact us via email at support at logsat.com we'll see how to proceed. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |