gmail greylisting |
Post Reply |
Author | |
dave.m
Newbie Joined: 05 August 2009 Status: Offline Points: 5 |
Post Options
Thanks(0)
Posted: 05 August 2009 at 1:44pm |
We recently downloaded SpamFilterISP and are currently doing tests with it. We are a website design company that hosts sites and e-mails for several companies and would like to migrate off of our existing spam solution that uses sendmail/spamassassin/etc on a Linux platform.
Installation went fine, and some tests have come through successfully. We're having an issue with gmail clients being able to send mail to us. I looked through the forums, and saw a couple of other posts related to this, and it appears to have to do with greylisting. If I understand correctly, when a mail comes in from gmail if the IP isn't on the greylisting "whitelist" it's told to check back in X minutes determined the greylisting setting. Apparently, gmail does try back at whatever time interval the gmail servers are set to try back at - but is most likely coming from a different IP address. Eventually this mail gets bounced back enough that it doesn't come through and the sender receives an undeliverable e-mail. We would like to know if anyone has found a way to to whitelist all gmail sending IP's. I know it can't be done by ranges from a previous post, but is there a way to import these into the database perhaps? Or has anyone else found a solution to this issue? Thanks, Dave |
|
LogSat
Admin Group Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
Post Options
Thanks(0)
|
Dave,
If you have a list of gmail's IPs, you can prevent specific IPs from being blocked by the GreyList filter as follows. SpamFilter stores the IPs that have been allowed to pass the greylist filter in the file:
\SpamFilter\Domains\GreyListAllowed.txt The entries in that file are in the form: aaa.bbb.ccc.ddd~nnnnn.ddddddddddd where aaa.bbb.ccc.ddd is the IP address, and nnnnn.ddddddddddd is a decimal date value. For example: 80.69.70.3~39495.5492583796 The decimal date value indicates the number of days that have passed since 12/30/1899. The fractional part of the value is fraction of a 24 hour day that has elapsed. SpamFilter will delete daily IPs whose date indicated above older than the number of days indicated in the following SpamFilter parameter (60 days by default): GreyListAllowedHold=60 You can manually insert the IP you want in the GreyListAllowed.txt file, and assign it a date that is a few years in the future. For example, adding 10 years would require adding 365x10=3650 to the value indicated: 80.69.70.3~39495.5492583796 plus 3650= 80.69.70.3~43145.5492583796 So, in short, simply add the above line to the GreyListAllowed.txt file. Please note that SpamFilter will only load this file during startup, it does not support changes made to it while it is running. You thus need to stop SpamFilter before making any manual changes to it. |
|
dave.m
Newbie Joined: 05 August 2009 Status: Offline Points: 5 |
Post Options
Thanks(0)
|
Thanks for the quick response.
So I guess our options are to add the 65,000 IP addresses that gmail supposedly sends from: 216.239.32.0/19
|
|
LogSat
Admin Group Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
Post Options
Thanks(0)
|
Correct - the greylist filter does not support subnets/wildcards. The reason for this is that the greylist filter works on a TCP level by blocking IPs directly and immediately upon connections. It must be very fast to avoid DDOS attacks, and the most efficient way we could implement it was by having a plain text list in RAM to perform fast lookups. Mixing the list with subnets would cause the lookups to be much slower. Of course adding 65,000 IPs will also make it slower... but not as much as having admins enter 100s of subnets mixed with thousands of IPs.
There are however very, very good admins on this forum, so if anyone has any opinions on this we always listen!
|
|
WebGuyz
Senior Member Joined: 09 May 2005 Location: United States Status: Offline Points: 348 |
Post Options
Thanks(0)
|
Roberto,
Understand you want peak performance, but the reality is that we need to be able to compensate for these large farms of smtp gateways, as well as handling greylisting sharing amongst multiple SFE's. I had to setup a non-greylisting version of SFE for a number of customer because of horrible delivery times because of the issue of greylisting and multiple IP's from a single large ISP. That really hurts when customers threaten to leave because of slow delivery when I bust my butt to deliver the best service possible. I would gladly give up performance to be able to not have this happen. What good is performance when your service is called poor because it can take an hour or more for an email to get from point A to B. Customer could care less that I have a high performance spam engine, all he knows is that his business contact who was on the phone with him sends him an email, but my customer does not get it until over an hour later, he calls to tell me my service sucks.
|
|
http://www.webguyz.net
|
|
Wayne
Groupie Joined: 29 August 2006 Location: Switzerland Status: Offline Points: 60 |
Post Options
Thanks(0)
|
Roberto
I totally agree with WebGuyz! We need this option and I'm also willing to sacrifice performance for that. I cannot believe that this whitelist implementation would took about an hour for most customers. Otherwise i cannot explain how the other anti-spam manufacturers have solved this problem and you know, they have. Regards Wayne Edited by Wayne - 10 August 2009 at 7:21am |
|
SF4.5.0.1-beta
|
|
yapadu
Senior Member Joined: 12 May 2005 Status: Offline Points: 297 |
Post Options
Thanks(0)
|
I often feel the delay of having to wait. I know every time I sign up for an online service I am going to have to wait an extended period of time... knowing the email will come in 15 minutes or maybe an hour (or sometimes the next day).
Greylisting is doing what it is supposed to be doing though, just having it on the server keeps the CPU load levels down quite a bit by keeping the trash connections from even getting to the server. I suspect there may be a suitable comprise, such as allowing individual IP addresses as well as a class C. That reduces the number of entries people have to make and should still keep the complexity of the lookups down. I don't know how SpamFilter works internally, but I have noticed that IP Addresses are all stored in 123.456.789.123 format in the databases. When dealing with IP addresses I find it much quicker to store them as a numeric base 10 value - some binary format is probably much better than that. Roberto is the master, and knows what spamfilter is doing behind the scenes but maybe storing the values in a different format would allow a speed improvement enough to allow some additions of range based lists. Another option might be to maintain two lists in memory for the greylists, one of just IP's and another of class C network ranges etc. You just need to search two lists, and you know the format of the addresses in both so mixing of IP's vs. Class C list searches should not cause big problems. I also vote for some type of shared list among servers. If you have a cluster of spamfilter servers the issue of inbound greylisting gets even worse, causing even longer delays. Imagine a cluster of 4 servers, some external server connects to server 1, gets rejected. The next time it tries it connects to server 2, then a different server from their cluster tries and gets server 3. Could take a very long time before the message gets through. I'm happy with the grey listing at the moment though. |
|
yapadu
Senior Member Joined: 12 May 2005 Status: Offline Points: 297 |
Post Options
Thanks(0)
|
Again, I'm just thinking out loud here... If Roberto is storing the addresses in memory as strings, there could be quite a bit of improvement if the addresses were converted to base 10. Then you could use some quick search methods, such as the discussion here:
http://leepoint.net/notes-java/algorithms/searching/binarysearch.html |
|
LogSat
Admin Group Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
Post Options
Thanks(0)
|
WebGuyz, When you mention "performance", am I interpreting correctly in that you'd prefer to loose some performance in exchange for the flexibility of being able to manually add subnets in the greylist? In regards to the sharing of the greylist amongst multiple servers, this is a *much* bigger challenge to address. Just to get the ball possibly rolling, could you let me know how many entries you currently have in the GreyListAllowed.txt file?
|
|
WebGuyz
Senior Member Joined: 09 May 2005 Location: United States Status: Offline Points: 348 |
Post Options
Thanks(0)
|
Our greylist file has about 235K entries and service about 5k mailboxes. We have 2 SFE's.
<soapbox>
Greylisting is a excellent tool and really has decreased the amount of traffic hitting our servers, but if it causes long delivery delays and causes perception that your service is 'slow' because sometimes it takes over an hour for an email to get delivered (because it was bouncing around between different IP or in our case multiple SFE's), then it becomes a liablilty.
We have an organ donation firm that needed timely emails from all around the country, but even when they whitelisted everyone they would still get some emails over an hour old because of the greylisting. They threatened to leave so I put them on a copy of SFE with no greylisting. Since then I've had to put 6 other companies on that same SFE with no greylisting because of complaints of slow delivery of emails, usually from senders in companies like IBM and other large corps that have a LOT of different IP's used to deliver mail and which they alternate retries.
I hate it when people tell me my service sucks and I can't do anything about it. I love SFE, but greylisting is its achilles heel in a multi SFE environment and where its not easily possible to put complete subnets in for accepting emails from large ISP's.
</soapbox>
|
|
http://www.webguyz.net
|
|
yapadu
Senior Member Joined: 12 May 2005 Status: Offline Points: 297 |
Post Options
Thanks(0)
|
My greylist file on the primary server has 200k, and the backup server has 66k. These have all been automatically added, we have not put in any manual entries.
|
|
Wayne
Groupie Joined: 29 August 2006 Location: Switzerland Status: Offline Points: 60 |
Post Options
Thanks(0)
|
So it seems I'm a little smaler, I only have 22k entries Therefrom are 200 manually added. |
|
SF4.5.0.1-beta
|
|
LogSat
Admin Group Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
Post Options
Thanks(0)
|
Within the next 24/36 hours we'll have an early beta for SpamFilter (both Standard and Enterprise) that allows the entry of IP wildcards (ex. 192.168.255.* or 192.168.*) in the GreyListAllowed.txt greylist file. You may contact us at support at logsat.com if you wish to test it before it's released on our website (please include your order number in the email so we can verify the current software maintenance).
|
|
WebGuyz
Senior Member Joined: 09 May 2005 Location: United States Status: Offline Points: 348 |
Post Options
Thanks(0)
|
You know what else would be really good, a way to do a soft reset to reload the greylist file without having to do a complete restart. The same way the spamfilter.ini is read in every minute, but it wouldn't have to be that frequently.
I know, I know, we keep asking for the sky, but when you have over 400 domains and a huge autowhitelistdelivery table when using SFE it take almost 5 minutes to load SFE into memory.
Thanks! Edited by WebGuyz - 13 August 2009 at 12:12am |
|
http://www.webguyz.net
|
|
Wayne
Groupie Joined: 29 August 2006 Location: Switzerland Status: Offline Points: 60 |
Post Options
Thanks(0)
|
Great news Roberto! Thanx!! |
|
SF4.5.0.1-beta
|
|
Desperado
Senior Member Joined: 27 January 2005 Location: United States Status: Offline Points: 1143 |
Post Options
Thanks(0)
|
Thought .... are we using the "dnsbl white list"? Can that be used to generate entries in the gray-list allow list that expire in a year or so?
Just letting ideas rattle around in my head!
|
|
The Desperado
Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
|
dave.m
Newbie Joined: 05 August 2009 Status: Offline Points: 5 |
Post Options
Thanks(0)
|
Roberto,
We're currently testing the software (see original post :)) Would like to know if we can get a copy of this beta so I can try and get it approved for purchase. Thanks! Dave |
|
LogSat
Admin Group Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
Post Options
Thanks(0)
|
Dave,
Yes, it's not going to be a problem. If you can you please contact us via email at support at logsat.com we'll see how to proceed.
|
|
Post Reply | |
Tweet
|
Forum Jump | Forum Permissions You cannot post new topics in this forum You cannot reply to topics in this forum You cannot delete your posts in this forum You cannot edit your posts in this forum You cannot create polls in this forum You cannot vote in polls in this forum |
This page was generated in 0.234 seconds.