webguy,

I had previously requested this in a direct message to LogSat:

"Can we come up with a way for SF Users to remove blocks in the SFDB from an IP other that the mail server IP?  I managed to get a hotmail server on the list with a mistake and want to clear it.  I can see this happening semi-often.

Stephen,

Or worse ... I see a lot of "allow everything" records.

I believe that the major players post their mailserver blocks somewhere on their websites. I think that only regional ISP's should qualify to have thier mail server IP blocks whitelisted. Maybe a list of regionals should be drawn up and people could volunteer to find a source of info for one or more of the ISP's mail server and forward the URL to SF support and they could verify it and add them. That way we could break up the workload load and feed the info to one source who has the power to whitelist without that person having to do all the legwork. A vote for some of the regionals are:

aol.com
msn.com
yahoo.com
gmail.com
earthlink.com
mindspring.com
netscape.net
netzero.com

I didn't say anything about SPF. I'm saying get the actual IP address of their outgoing mail servers.

 

ok, let me explain my thoughts about SPF and the topic.

First of all, the easiest (and may be the only right) way to obtain foriegn ISP's mailservers IPs is (or should be) SPF. You may only wish that you know all their MS IPs, but for large organizations this list is very big and dynamicly changing. So (uless you use SPF) you would have to constantly _manually_ monitor them. With clear SPF _pass_ answer you must be sure that certain IP is allowed to mail certain DomainFROM.

Therefore, WebGuyz, I'm not sure that we could "get the actual IP address of their outgoing mail servers" if they are lazy enough to not support even correct SPF record.

Otherwise, Dan is right about all the problems. My original idea was to (optionally) reward "-all" records, but rules "ip4:0.0.0.0/0" could ruin this too...

Roman,

Your comment "So de facto I should use "softfail" and "neutral" responses to mark spam. I know thats not good, but ...."  Is a problem that I flip/flop on.  The SPF proposal states that you should allow these but can use the results to "score" Spam.  I do allow them but get hammered as a result.  I really wish that ISP's (such as ourselves) would take responsibility for their mail server IP's.  Is AOL going to try to convince me that they don't know for sure what IP's they are using?  Or ... is it a hedge ... I mean, they are one of the bigest pushers of SPF and their own record is ?all  Something wrong here.

SPF is not a good choice because of the reasons listed above. But why not list all the IP blocks owned by aol.com, yahoo.com, etc.

These can be found by checking my logs for some current IP addresses and verifying them in a tool like Sam Spade and where I can get a list of the IP blocks owned by the ISP.

Don't lose site of the fact that we are not whitelisting them by adding the ip blocks to SFDB, we are just relying on all the other SF tests as we have in the past to avoid false positives. My stats show that the highest stopping filter on my server is SFDB and thats great, but I don't want someones filter updating SFDB with an IP causing me to stop being able to receive from the largest ISP's. This has only happened once, but I can see it happening again.

SF is tuned over time to filter spam for your users, Adding SFDB to the mix is good but some of that filter uniqueness (is that a word?) is lost and by blacklisting some of the big guys  I am at the mercy of someones unique spam fighting tactics.

Desperado wrote: 13. Keywords found in content - NO  Some people WANT or even NEED to enlarge their body parts!

...Interesting, but WAY more than I needed to know about you. 

Stephen

Stephen, there is no "legitimate reason for creating softfail and neutral spf rules" (in long term perspective)!

First of all (if I remember correctly) origionally "?" is for "testing" and "~" is for domains "where not all clients are upgraded to use correct smtp relays yet". But "smart" guys decided: "I want to participate in antispam and I want 100% deliverability, but I am very lazy... What can I do? Oh! There goes "~/?" !". This sick method is even allowed by http://www.openspf.org/whitepaper.pdf . So we have the illusion that big ISPs support antispam policy, but de facto it is ABSOLUTELY USELESS (if follow original standards). Well, if the biggest ISPs use this sick method I'm constrained to block "?/~". Anyway, what should "a:some ?all" mean? As I can see it mean "host "some" is ok to send mail, and I am in testing phase for all other IPs in the world". When are you gonna finish your endless global testing??? Sorry, this is defenetly not right.

About your particular case. Spam became the huge problem because of insecure smtp standard. But now with current "best practicies" (where the hell new RFC is ?!?) it is a problem only besause man's laziness and incompetence - spammers can only relay their crap through WRONGLY configured servers. So administration task became very clear - do everything RIGHT and block everything WRONG. By assuming that your hosting company could fail in RIGHT authentication of customers and therefore using "?" you actually say: "my hoster (presumably) has WRONG configuration in his servers, so I should also implement WRONG configuration on my side". Then please don't be disappointed when I block you.

If you use anyone's relay, please, make it "pass", make it RIGHT. Yes, if he would fail in authentication (or anything else) - he will be blocked because he did something WRONG. And then he will have a chance to do it RIGHT.

Dan, about your SFDB usage policy (as long as opinions are welcomed :) . Consider this:

The main idea of SFDB is to mark WRONG relays. Let's look at 1st point. People usually (I mean IMHO. Myself.) block some domain, let's say bonbon.net not because they are/were subscribed to their mailing lists, but because spam/viruses fakes the return addr.. In other words I block some domain only if
1. SpamHits number is more than <some predefined number>
2. The possibility of real business with it's owners is zero (and I have never seen their legitimate mail ever)
3. It has no SPF record (actually my custom error for this rule sounds like: "550 Your domain %Domain% does not have SPF record and/or is Blacklisted...")
So when I mark some server for "some@bonbon.net" there is no chance that I'd mark the REAL bonbon server - it always will be some stupid ADSL user. So it's pretty safe to use (not much or less than others).

The same applies to 5, 6 (in my config - never => I use this rule => only spammers will be marked).
May be true also for 13 and 14 (not 100% sure, needs research).
17 - always bad|stupid boy - must be marked.
20 - well, very strange for me to see this rule at all - "autoinflate" possible.
3, 15, 16 you can do it yourself as 12 - see no reason.

Edited by Roman

Roman, I understand that you can have your own unique interpretation of SPF results in order to curb spam, but your interpretation doesn't match up to what's in the spec.  True, neutral and softfail results are to be used in the testing phase, but that is not their only usage.

From http://new.openspf.org/svn/project/specs/rfc4408.html#op-res ult:

2.5.2.  Neutral The domain owner has explicitly stated that he cannot or does not want to assert whether or not the IP address is authorized. A "Neutral" result MUST be treated exactly like the "None" result; the distinction exists only for informational purposes. Treating "Neutral" more harshly than "None" would discourage domain owners from testing the use of SPF records (see Section 9.1 (Sending Domains)). ... 2.5.5.  SoftFail A "SoftFail" result should be treated as somewhere between a "Fail" and a "Neutral". The domain believes the host is not authorized but is not willing to make that strong of a statement. Receiving software SHOULD NOT reject the message based solely on this result, but MAY subject the message to closer scrutiny than normal. The domain owner wants to discourage the use of this host and thus desires limited feedback when a "SoftFail" result occurs. For example, the recipient's Mail User Agent (MUA) could highlight the "SoftFail" status, or the receiving MTA could give the sender a message using a technique called "greylisting" whereby the MTA can issue an SMTP reply code of 451 (4.3.0 DSN code) with a note the first time the message is received, but accept it the second time.

...Neutrals aren't just used by lazy admins.  I think that my case - where I used a shared hosting company that has both good and bad customers is a perfect example of where a customer such as myself would hesitate to stamp certain emails as a "pass", when same smtp server sends unsolicited email for abusive customers.

I've tested my web hosting company's smtp server, and confirmed that it is not an open relay.  By getting on AOL's FBL (Feedback loop), I was able to receive a copy of all spam sent out through my hosting company.  The mail headers helped me identify that the outgoing spam is coming from actual customers from the web hosting company, who are using their web sites to send out spam.

Notice that I do use a -all in my SPF record, it is only email originating from the shared web hosting company that I use that I mark as neutral.

v=spf1 +mx:mail.mydomain.com ?mail.sharedhostingcompany.com -all

If I were to avoid any neutral or softfail results, as you suggest, then I would only have the following options - none of which are acceptable to me:

SPF Record                                                    Consequence +mail.sharedhostingcompany.com -all   Spammers on my hosting company's                                        site can send mail on my domain's                                        behalf. My SPF record asserts that                                        the email is from me. -mail.sharedhostingcompany.com -all   All email from my web hosting                                        company's web site (including the                                        email that I send) is marked as                                        forgery. (Deleting my SPF record)        I lose all benefits from SPF                                  validation, in efforts to make sure                                  that you can receive my mail.

I do agree with you - that neutrals and softfails are widely overused by lazy administrators.  Unfortunately, the Big Boys (Hotmail, Google, etc..) are some of the worst offenders.  But as you consider keeping your SPF acceptance policy more aggressive, I hope you consider that my isolated use for neutral results from one of my outgoing email servers is appropriate.

Stephen

Edited by sgeorge

Roman, along the idea of prospective enhancements in SpamFilter’s SPF filtering options, wouldn’t it be nice to be able to explicitly override an external domain’s SPF record with one of your own making?

This would allow finer-grain control to tightening lazy SPF records.  We could make better use of the lazy SPF records from the Big Guys, without rejecting any and all responses of neutral and softfails.

If one could copy and paste Gmail’s existing SPF record into SpamFilter, changing the useless ?all to a –all, for example, one could stand to stop a lot of spam.

I suppose though, that the right rejection code would have to be used.  Returning an error due to the “Sender’s” policy would not be accurate.

Stephen

interesting thread.

we switched to having soft fails blocked due to the amount coming from hotmail that got through on this check.

it may not be perfect but we had 100s getting through of hard to block spam... with the config change, all are blocked.

it would be nice if the soft fail option was configurable per domain... or something we could override so we can get the benefits of both options and a bit more granularity.