I was looking through logs to see why a message may not have been flagged as spam.

I found this snippet interesting:

03/31/20 10:49:42:912 -- (1140421328) Resolving 185.236.202.242 - no-mans-land.m247.com

03/31/20 10:49:43:109 -- (1140421328) - SPF analysis for dhl.com done: - none
03/31/20 10:49:43:109 -- (1140421328) Mail from: Invoice@dhl.com
03/31/20 10:49:43:125 -- (1140421328) - MAPS search done...

The 1st line leads me to believe that the DNS server is working as it successfully resolves the reverse lookup.

The next line talks about SPF, attempting to resolve the SPF details for dhl.com, it says none.

Does that mean one of these?

1) There was no SPF record for dhl.com

2) The server returned an error?

3) The server got the SPF record but it passed?

4) Something else?

There was no other mention of SPF related to this message.

A search prior in the logs (different messages) shows:

03/31/20 10:49:40:723 -- (1878202896) - SPF analysis for dhl.com done: - softfail

03/31/20 10:50:05:566 -- (1140421328) - SPF analysis for dhl.com done: - softfail

Both seconds before and seconds after when processing different messages there was some result.

yapadu,

This is looking like a bug that is causing the SPF test to be skipped sometimes. I will send you a separate email so as to not make public the issue until we resolve it.

Thanks for finding and reporting it!

ik8sqi, did you ever find a fix for DHL?

If not, I'll just give them a call and have them reduce the number of TXT records they publish ;-)

This one is being rather challenging... The TXT record being returned by DHL is over 512 bytes in size (1155), which is DNS's default max message size in UDP (per RFC 1035). There is an extension mechanism in DNS (EDNS - RFC 6891) that allows for 1280 bytes instead of 512, which would barely be enough to handle DHL's... but SpamFilter does not support that extension.

DNS clients that do not support the EDNS extension would normally re-issue the DNS query using TCP instead of UDP. TPC-based DNS queries does not contain that 512 bytes limitation... but SpamFilter's DNS implementation is not behaving correctly and is not reverting to TCP.

All this to say that our implementation of the DNS client is flawed, and upgrading it requires some major internal upgrades which will take some time to complete - this is not turning out to be one of those easy bug fixes which we're often able to release, sorry!

Oh, that sounds nasty and possibly not worth fixing due to other complications it could cause.  It is probably very rare that a domain has such a large TXT record.

Some DNS servers will not allow queries via TCP either, so that could be another issue in itself.

I was thinking about other ways to possibly work around the problem.

How about caching TXT records for a while?  Even if you cached the TXT SPF records for a domain (based on their TTL?) you would quickly populate your cache once the TXT was returned within the byte limit. 

If you try and refresh your cache at 50% of the TTL you prevent ever having the cache expire, which would introduce the bug again until the cache populated itself.  I think DHCP does something like this, try and renew before actual expiration.

If a domain has no TXT record, or there was an error obtaining it you don't cache that (which is what SF does now).

A cache would boost SF performance as well.

Even if I cached the TXT records, in order to populate the ones over the 512 bytes which would eventually randomly be retrieved, this means we'd have to have a process of checking the cache, making a DNS query if no SPF record found, and then if found add it to the cache, and then cleanup this cache after sometime.. it's getting really ugly and complicated.

I'm still trying to find a "real" fix for this -have not given up yet.