Maybe exclude aol.com and other IP’s from
Printed From: LogSat Software
Category: Spam Filter ISP
Forum Name: Spam Filter ISP Support
Forum Description: General support for Spam Filter ISP
URL: https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=5621
Printed Date: 27 December 2024 at 2:42pm
Topic: Maybe exclude aol.com and other IP’s from
Posted By: WebGuyz
Subject: Maybe exclude aol.com and other IP’s from
Date Posted: 20 May 2006 at 10:31am
|
Hi, I think its not practical to block aol.com, hotmail.com, gmail.com valid IP blocks in the SFDB bacause of the large volume of mail and the potential of anyone of them being blacklisted and added to the SFDB causes more issues than its worth. Anyone else have an opinion? ------------- http://www.webguyz.net |
Replies:
Posted By: Desperado
Date Posted: 20 May 2006 at 4:38pm
|
webguy, I had previously requested this in a direct message to LogSat:
Also, How about a SFDB whitelist such as aol has?"
I think something should be looked at but I am not sure what. As an ISP, we went through a lot to get whitelisted by aol but have to deal with nearly hourly reports from them. Hotmail ... well they won't play so I am not real happy with them anyway. Other thoughts?
------------- The Desperado Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
Posted By: Roman
Date Posted: 23 May 2006 at 11:51am
|
my 2 cents...
May be it could be reasonable to (optionally?) skip SFDB checks for IPs which PASS SPF test? |
Posted By: Desperado
Date Posted: 23 May 2006 at 12:46pm
|
Hmmmm I suspect that that would drastically reduce the effectiveness of SFDB. Need to think more on that. Besides, With AOL, you need now to define "pass" as their own SPF record is loose:
Here are some general thoughts ... ------------- The Desperado Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
Posted By: sgeorge
Date Posted: 23 May 2006 at 1:23pm
|
Dan, on that note of uselessly loose SPF
records, I was just thinking today that it would be great to be able to
override a domain's useless SPF rules. Some may have they're reasons for
not getting their acts together yet, but with all of the PayPal scams out there
these days (for one example), I'm amazed that they still have not been able to
publish a -all SPF record to date for themselves. Stephen |
Posted By: Desperado
Date Posted: 23 May 2006 at 2:46pm
|
Stephen, Or worse ... I see a lot of "allow everything" records. ------------- The Desperado Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
Posted By: LogSat
Date Posted: 23 May 2006 at 6:48pm
|
The SFDB has the capability of "whitelisting" IP addresses. The process on how this is to be done is yet to be defined. We have thought about allowing licensed users whitelist IPs from the registered user area, but that could lead users whitelisting spammer's IP (either on purpose or by mistake...). Allowing the owenrs of the blacklisted IPs to whitelist themselves is also a bad idea... We are open to suggestions! ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: WebGuyz
Date Posted: 23 May 2006 at 7:47pm
|
I believe that the major players post their mailserver blocks somewhere on their websites. I think that only regional ISP's should qualify to have thier mail server IP blocks whitelisted. Maybe a list of regionals should be drawn up and people could volunteer to find a source of info for one or more of the ISP's mail server and forward the URL to SF support and they could verify it and add them. That way we could break up the workload load and feed the info to one source who has the power to whitelist without that person having to do all the legwork. A vote for some of the regionals are: aol.com ------------- http://www.webguyz.net |
Posted By: Desperado
Date Posted: 23 May 2006 at 8:42pm
|
Webguyz,
What you are saying is you "Trust" the "big boys". Well, I don't. If we are to take your tact, I would weight it against their SPF record (mentioned earlier but someone).
aol.com Loose SPF with a ?all
msn.com Also loose with hotmail includes and a ~all
yahoo.com No SPF at all
gmail.com Your kidding right? Redirect to google.com with a ptr ?all I can spoof that is a heart beat
earthlink.com No SPF at all
mindspring.com No SPF at all
netscape.net No SPF at all
netzero.com Loose SPF with a ?all
What does this say about their commitment to anti-spam? Of the above list, aol is the ONLY one we have had any real success communicating with about abuse/spam either to or from our network. They have worked with us very closely as we do have customers on our network doing huge mailings. Also, we went to Chicago for a "Email deliverability Boot Camp" and aol seemed to have the brightest people there and had the most reasonably implementable (is that a word?) ideas. msn/hotmail simply wanted to make money guaranteeing mail deliverability (big surprise there) and Yahoo .... let's just say that their name describes my opinion of their representative.
Aren't I just a cranky, cynical SOB?
SO ... my thought ... A list of "Trusted" Mail servers should only include servers with SPF records and should be SEMI-Whitelisted. Meaning, do not add to the SFDB until "X" number of submissions come in and expire them faster ... say 1-4 hours, Kinda like aol does their black-listing. ------------- The Desperado Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
Posted By: WebGuyz
Date Posted: 23 May 2006 at 9:11pm
|
I didn't say anything about SPF. I'm saying get the actual IP address of their outgoing mail servers.
------------- http://www.webguyz.net |
Posted By: Desperado
Date Posted: 23 May 2006 at 9:26pm
|
Webguys, I understand that BUT, I do not want to simply trust their mail servers. Inclusion of an SPF record to a domain shows at least an attempt to join the anti-spam community so I am using SPF as a starting point to start trusting them. If a mail provider makes no attempt to prevent their addresses from being spoofed, how can I believe they will stop their users from spamming? And if they do not communicate easily to another ISP concerning spam issues ... same problem. That was all my point was. ------------- The Desperado Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
Posted By: Roman
Date Posted: 24 May 2006 at 10:10am
|
ok, let me explain my thoughts about SPF and the topic.
First of all, the easiest (and may be the only right) way to obtain foriegn ISP's mailservers IPs is (or should be) SPF. You may only wish that you know all their MS IPs, but for large organizations this list is very big and dynamicly changing. So (uless you use SPF) you would have to constantly _manually_ monitor them. With clear SPF _pass_ answer you must be sure that certain IP is allowed to mail certain DomainFROM. Therefore, WebGuyz, I'm not sure that we could "get the actual IP address of their outgoing mail servers" if they are lazy enough to not support even correct SPF record. Otherwise, Dan is right about all the problems. My original idea was to (optionally) reward "-all" records, but rules "ip4:0.0.0.0/0" could ruin this too... |
Posted By: Roman
Date Posted: 24 May 2006 at 11:00am
|
and some offtopic thoughts about SPF.
The situation with major IPSs went absolutely crazy. The fact is that (almost) all of them use "?all" (or "~all") records. So de facto I should use "softfail" and "neutral" responses to mark spam. I know thats not good, but that is the only way stop junk from google and hotmail (the 1st and the 3rd in my spam domains top). The 2nd place for spam is yahoo... I don't know what to say. May be DomainKey is absolutely perfect, but why the hell not to use SPF? I can only remember russian KGB joke: "if you want to sabotage dissident organization - lead it". |
Posted By: Desperado
Date Posted: 24 May 2006 at 11:23am
|
Roman, Your comment "So de facto I should use "softfail" and "neutral" responses to mark spam. I know thats not good, but ...." Is a problem that I flip/flop on. The SPF proposal states that you should allow these but can use the results to "score" Spam. I do allow them but get hammered as a result. I really wish that ISP's (such as ourselves) would take responsibility for their mail server IP's. Is AOL going to try to convince me that they don't know for sure what IP's they are using? Or ... is it a hedge ... I mean, they are one of the bigest pushers of SPF and their own record is ?all Something wrong here. ------------- The Desperado Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
Posted By: sgeorge
Date Posted: 24 May 2006 at 12:08pm
In the spirit of derailing this topic train further off course...  I would disagree with the policy of universally blocking softfail and/or neutral spf results. It may give you great results in blocking spam, but it punishes anyone who happens to have a legitimate reason for creating softfail and neutral spf rules. What's this? Could there actually be a practical reason for "softfail" and "neutral". I would say that there is... In fact, I do use it - to the least extent that I have to. My domain's spf 1 record looks something like this:
...The story is that email that legitimately can be from "...@mydomain.com" comes from two sources... 1.) my private mail server, or 2.) my web site - which is on a shared web host. This shared hosting company is completely inept when it comes to tracking down customers of theirs that send out spam. And yhe last thing I need is for one of these spamming customers to get smart enough to fake a peer customer's email address - knowing that the peer's spf record is likely to have a "pass" rule for all email originating from the shared hosting company. Effectively, my spf record would assert that spam alleged to be from @mydomain.com really is. So, yes, I find that making a neutral rule for my shared web host helps me distance myself from peer spammers, while preventing the outgoing email from my web site by getting kicked by my -all. My thought is is this... the more we push our spf verification criteria past design, the less likely we are to get the big boys to play nice. True, they will get the complaints from their customers when mail isn't delivered. But what do you really think their response is going to be? Make a -all record? I bet that a +all is more likely... Stephen |
Posted By: Guests
Date Posted: 24 May 2006 at 5:50pm
|
SPF is not a good choice because of the reasons listed above. But why not list all the IP blocks owned by aol.com, yahoo.com, etc. These can be found by checking my logs for some current IP addresses and verifying them in a tool like Sam Spade and where I can get a list of the IP blocks owned by the ISP. Don't lose site of the fact that we are not whitelisting them by adding the ip blocks to SFDB, we are just relying on all the other SF tests as we have in the past to avoid false positives. My stats show that the highest stopping filter on my server is SFDB and thats great, but I don't want someones filter updating SFDB with an IP causing me to stop being able to receive from the largest ISP's. This has only happened once, but I can see it happening again. SF is tuned over time to filter spam for your users, Adding SFDB to the mix is good but some of that filter uniqueness (is that a word? |
) is lost and by blacklisting some of the big guys I am at the mercy of someones unique spam fighting tactics.Posted By: Desperado
Date Posted: 24 May 2006 at 7:00pm
|
OK ... I am going to take a different tact here. WHY are the "Big Boys" getting on the SFDB in the first place? As Webguyz stated (And yes that is a word!), "I am at the mercy of someones unique spam fighting tactics." I see that at the most accurate reason for the blacklisting of the above mentioned "Big Boys". HOWEVER, I use the SFDB feature of only accepting SFDB uploads that I feel are the least subjective to, for the most part, solve that issue. Consider the following and note that this is MY opinion and MY method only:
SFDB Checks for IPs blocked by selected filters:
Thoughts and opinions always welcome. ------------- The Desperado Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
Posted By: sgeorge
Date Posted: 25 May 2006 at 1:10pm
...Interesting, but WAY more than I needed to know about you.  Stephen |
Desperado wrote:Posted By: Desperado
Date Posted: 25 May 2006 at 1:19pm
|
Stephen, Roger that! ------------- The Desperado Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
Posted By: Roman
Date Posted: 31 May 2006 at 11:57am
|
Stephen, there is no "legitimate reason for creating softfail and neutral spf rules" (in long term perspective)!
First of all (if I remember correctly) origionally "?" is for "testing" and "~" is for domains "where not all clients are upgraded to use correct smtp relays yet". But "smart" guys decided: "I want to participate in antispam and I want 100% deliverability, but I am very lazy... What can I do? Oh! There goes "~/?" !". This sick method is even allowed by http://www.openspf.org/whitepaper.pdf . So we have the illusion that big ISPs support antispam policy, but de facto it is ABSOLUTELY USELESS (if follow original standards). Well, if the biggest ISPs use this sick method I'm constrained to block "?/~". Anyway, what should "a:some ?all" mean? As I can see it mean "host "some" is ok to send mail, and I am in testing phase for all other IPs in the world". When are you gonna finish your endless global testing??? Sorry, this is defenetly not right. About your particular case. Spam became the huge problem because of insecure smtp standard. But now with current "best practicies" (where the hell new RFC is ?!?) it is a problem only besause man's laziness and incompetence - spammers can only relay their crap through WRONGLY configured servers. So administration task became very clear - do everything RIGHT and block everything WRONG. By assuming that your hosting company could fail in RIGHT authentication of customers and therefore using "?" you actually say: "my hoster (presumably) has WRONG configuration in his servers, so I should also implement WRONG configuration on my side". Then please don't be disappointed when I block you. If you use anyone's relay, please, make it "pass", make it RIGHT. Yes, if he would fail in authentication (or anything else) - he will be blocked because he did something WRONG. And then he will have a chance to do it RIGHT. |
Posted By: Desperado
Date Posted: 31 May 2006 at 12:18pm
|
Roman,
That was a little reminiscent of "Who's on first?"
In principal, I really do agree with your statements and I get very irritated at servers and DNS that are configured wrong or sloppily (Wow - Spell check says that's a word). However, sometimes, it is not laziness but extreme pressure from our remote customers who don't give a rat's you know what about "right" or "wrong" and just want their mail to go through. Also, some of our own sales force want their Sprint PCS to work and their IP is dynamic and even crosses the /24 barrier so what do we do there? I have changed my SPF record over and over again to make up for that. We tried to get the stupid "Smart Phones" to use smpt auth and it fails about 30% of the time (??). I do not have a good answer but I do have a -all in my SPF. I just have to live with the problems it causes if I am claiming to be an anti-Spam crusader. ------------- The Desperado Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
Posted By: Roman
Date Posted: 31 May 2006 at 12:44pm
|
Dan, about your SFDB usage policy (as long as opinions are welcomed :) . Consider this:
The main idea of SFDB is to mark WRONG relays. Let's look at 1st point. People usually (I mean IMHO. Myself.) block some domain, let's say bonbon.net not because they are/were subscribed to their mailing lists, but because spam/viruses fakes the return addr.. In other words I block some domain only if 1. SpamHits number is more than <some predefined number> 2. The possibility of real business with it's owners is zero (and I have never seen their legitimate mail ever) 3. It has no SPF record (actually my custom error for this rule sounds like: "550 Your domain %Domain% does not have SPF record and/or is Blacklisted...") So when I mark some server for "some@bonbon.net" there is no chance that I'd mark the REAL bonbon server - it always will be some stupid ADSL user. So it's pretty safe to use (not much or less than others). The same applies to 5, 6 (in my config - never => I use this rule => only spammers will be marked). May be true also for 13 and 14 (not 100% sure, needs research). 17 - always bad|stupid boy - must be marked. 20 - well, very strange for me to see this rule at all - "autoinflate" possible. 3, 15, 16 you can do it yourself as 12 - see no reason. |
Posted By: Roman
Date Posted: 31 May 2006 at 12:54pm
|
"but extreme pressure from our remote customers who don't give a rat's" - that's another point: WE NEED RFC.
Well, I can only suggest to move obsolete (good adj. for SmartPhones :) equipment to subdomain with wide or no SPF... Anyway, as long as "big guys" use "?~all" I see no difference between "~?-all" at all. |
Posted By: sgeorge
Date Posted: 31 May 2006 at 1:46pm
|
Roman, I understand that you can have your own unique interpretation of SPF results in order to curb spam, but your interpretation doesn't match up to what's in the spec. True, neutral and softfail results are to be used in the testing phase, but that is not their only usage. From http://new.openspf.org/svn/project/specs/rfc4408.html#op-result - http://new.openspf.org/svn/project/specs/rfc4408.html#op-res ult :
...Neutrals aren't just used by lazy admins. I think that my case - where I used a shared hosting company that has both good and bad customers is a perfect example of where a customer such as myself would hesitate to stamp certain emails as a "pass", when same smtp server sends unsolicited email for abusive customers. I've tested my web hosting company's smtp server, and confirmed that it is not an open relay. By getting on http://postmaster.aol.com/fbl/index.html - AOL's FBL (Feedback loop), I was able to receive a copy of all spam sent out through my hosting company. The mail headers helped me identify that the outgoing spam is coming from actual customers from the web hosting company, who are using their web sites to send out spam. Notice that I do use a -all in my SPF record, it is only email originating from the shared web hosting company that I use that I mark as neutral.
If I were to avoid any neutral or softfail results, as you suggest, then I would only have the following options - none of which are acceptable to me:
I do agree with you - that neutrals and softfails are widely overused by lazy administrators. Unfortunately, the Big Boys (Hotmail, Google, etc..) are some of the worst offenders. But as you consider keeping your SPF acceptance policy more aggressive, I hope you consider that my isolated use for neutral results from one of my outgoing email servers is appropriate. Stephen |
Posted By: Roman
Date Posted: 31 May 2006 at 3:01pm
|
Stephen, as you wrote "neutral ... stated that [domain owner] cannot or does not want to assert whether or not the IP address is authorized". If you explicitly use some relay for your mail (I think) you should authorize that IP for this task. If that relay will fail to separate you from spamming (and faking your authorization) customer, well, may be it is not a very good relay. I’m afraid his IP (sooner or later) will be blacklisted if not for wrong SPF but for something else.
My point is: if you authorize some server to relay your mail then you should do it explicitly. You should not inherit foreign errors in your own configuration. I know, acc. to original meaning of SPF you are (maybe) right. But current practice of big ISPs makes it wrong (ok, only for me maybe). Well, I have a suggestion for Roberto which could help us all: To make an option to treat “~all”, “?all” (and maybe “+all”) as “-all”, but process explicit “neutral/softfail” responses as it does now. In other words I’d restrict all SPF statements which says “pass|neutral the world”. |
Posted By: sgeorge
Date Posted: 01 June 2006 at 11:38am
|
Roman, along the idea of prospective enhancements in SpamFilter’s SPF filtering options, wouldn’t it be nice to be able to explicitly override an external domain’s SPF record with one of your own making? This would allow finer-grain control to tightening lazy SPF records. We could make better use of the lazy SPF records from the Big Guys, without rejecting any and all responses of neutral and softfails. If one could copy and paste Gmail’s existing SPF record into SpamFilter, changing the useless ?all to a –all, for example, one could stand to stop a lot of spam. I suppose though, that the right rejection code would have to be used. Returning an error due to the “Sender’s” policy would not be accurate. Stephen |
Posted By: Guests
Date Posted: 15 June 2006 at 4:59pm
| I like the idea of having an SPF override to force a -all on some of these loose records. Not sure though what sort of worms that would let out of the can with regards to when they add/remove mail servers from their pool. |
Posted By: MartinC
Date Posted: 05 September 2006 at 10:12am
|
interesting thread. we switched to having soft fails blocked due to the amount coming from hotmail that got through on this check. it may not be perfect but we had 100s getting through of hard to block spam... with the config change, all are blocked. it would be nice if the soft fail option was configurable per domain... or something we could override so we can get the benefits of both options and a bit more granularity. |