Spam Filter ISP Support - Quarantine / Filter Order / Suggestion

Quarantine / Filter Order / Suggestion

Printed From: LogSat Software
Category: Spam Filter ISP
Forum Name: Spam Filter ISP Support
Forum Description: General support for Spam Filter ISP
URL: https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=7169
Printed Date: 27 December 2024 at 1:26am

Topic: Quarantine / Filter Order / Suggestion

Posted By: I.P.
Subject: Quarantine / Filter Order / Suggestion
Date Posted: 21 August 2020 at 2:00pm

i would like to integrate a spamfilter technique in spamfilter IPS that does not seem to work the regular way.

i want to put any mail to a defined recipient (Local Emails TO Blacklist) in the quarantine and then analyze the quarantine contents by an own software and release them or not.

i do not want to lose all different reasons why spamfilter ISP puts mails to quarantine because most of them are valid and close to 0 positive-false.

the problem is, that the Local Emails TO Blacklist steps in very early in the filter order and then the rejectid is always 9, even when other filters would quarantine the same email as well.

i need to seperate mails put into quarantine for Local Emails TO Blocklist from all other possible quarantine causes.

one possibility would be to put 2 spamfilters in serial and only the second one uses the Local Emails TO Blacklist filter, but this will be not too easy either.

Replies:

Posted By: ik8sqi
Date Posted: 22 August 2020 at 3:07pm

I.P.

I'm afraid there's not a good solution to your request. As you correctly said, when an email is sent to a blacklisted recipient, SpamFilter will immediately mark it as spam and for efficiency/performance reasons no more filters will be applied - the email will be blocked because of that match.

We've tested your idea of placing two SpamFilter inline - the first with all filters enabled except for the "Emails TO Blacklist", which then forwards the "clean" emails to the second SpamFilter, which instead must have all filters disabled except for the "Emails TO Blacklist". While this actually works, there's a big concerns with what happens when the second SpamFilter rejects the email because the "Emails TO Blacklist" is triggered.

This is how the email flow happens.

The first SpamFilter accepts the connection and checks the email for spam. During this entire process, SpamFilter places "on hold" the connection from the remote sender, and does not yet reply with a "250 OK" or another SMTP error. This is (unfortunately for you) actually one of the strengths in SpamFilter, because before replying to the sender we actually start the delivery of the email to the destination SMTP server. If, and only if the destination SMTP server accepts at least one recipient, then we are to believe the email will be accepted by the destination SMTP server, and at that point the first SpamFilter will issue a "250 OK" SMTP code back to the sender to let them know the email was delivered. This is usually a great feature, as if your destination SMTP rejected the recipient (ex. for a non-existent user, or a mailbox full) then SpamFilter would report the same exact rejection notice to the sender, which forces the sender, and not SpamFilter, to send an NDR back to the sender.

In your case however, since you want the second SpamFilter to quarantine the email if it's stopped by the Mail TO Blacklist, the second SpamFilter must accept the recipient as we need to receive the entire email to quarantine it. However, since the second SpamFilter accepted the recipient, the first SpamFilter then believes the email will be accepted, and will issue a "250 OK" to the sender to inform them it was delivered.

... but it wasn't delivered, as the destination SMTP server is not your real SMTP server (which we assume to always accept SpamFilter's emails), but rather it's another SpamFilter which will eventually reject the email. The first SpamFilter does see eventually that the email was rejected by the second SpamFilter, but it's too late as it already sent the "250 OK" to the sender's connection. So at this point, since the first SpamFilter wasn't able to deliver the email, we will have to send an NDR bounce email back to the sender. This will create backscatter, as for every email stopped by the 2nd SpamFilter, the 1st SpamFilter will send an NDR email back to the sender, which will often be a fake email, possibly belonging to a legitimate person who had their email address spoofed.

All this would not happen if the 2nd SpamFilter rejected the email without quarantining it, since the RCPT TO would not be accepted and thus the 1st SpamFilter would immediately see the rejection and terminate the connection with the sender.

We could resolve this by also having the 1st SpamFilter keep the connection from the sender "on hold" until the entire email is delivered to the destination SMTP server. But as usually the destination SMTP server for the 1st SpamFilter is your final mail server which would not check the email for spam, doing so would decrease performance and is something we've never looked into.

I'm sorry!

-------------
Roberto Franceschetti
LogSat Software

Posted By: I.P.
Date Posted: 24 August 2020 at 5:11am

Thanks for the answer Roberto,

i understand the 250 OK issue, but it does not seem like a big thing to me as i can route every local domain to different smtp servers on the first spamfilter and only domains that want to use this technique will be routed to the second spamfilter and being quarantined there.

i usually accept all mails on my mailservers and send no NDRs for nonexistant mailboxes, so i make no use of this feature.

is it possibly in some way to release quarantined mails in the quarantine to a different smtp server than the one defined for de recipient's domain?

the idea is:

spamfilter as MX - all filters exept BlacklistTO - then a second filter instance with BlacklistTO filter for some recipients and put the mails into spamfilters quarantine database with rejectid=9.

so i have only 1 quarantine per user he can self-manage.

the only problem to solve then is that released mails from quarantine have to be routed to a different server because the second spamfilter would quarantine them again.

additinal idea (and possible solution for this) - i set a custom mail header in the quarantine and whitelist this mailheader on the second spamfilter. but i would prefer routing the mails to a different host when released from quarantine. in fact i would appreciate this possibility in general, i opens the option as well to log mails that are released from quarantine by users.

Posted By: ik8sqi
Date Posted: 24 August 2020 at 6:26pm

There may actually be a solution if you're not concerned about the backscatter. But before describing it, I'd like to mention one last time the risk with the backscatter. The feature where SpamFilter puts the connection on hold while checking if a recipient is going to be accepted is enabled by default thru this setting in the SpamFilter.ini file:

;To avoid backscatter, if an incoming email passes all filtering rules, but cannot be forwarded (ex. mailbox full, non-existent user), SpamFilter maintains open the incoming remote connection until it can verify with the destination server that the email can be delivered. If not, a 5xx error is output forcing the remote server to generate the NDR, rather than having SpamFilter send an NDR notification email

VerifyRCPTTOForCleanEmails=1

So it's possible/likely you're already benefiting from it.

When someone will send emails to a blacklisted TO email address, this will cause an NDR email to be sent back to the fake "from" email address used by the spammer. It won't be an issue if the number of emails sent to those blacklisted TO addresses is small, but if the number is high you risk having your mail server's IP being blacklisted.

This said, to the possible solution. In installations where there are multiple SpamFilter servers all using the same quarantine database, each SpamFilter will know which emails it "owns" so that that specific SpamFilter will be the one responsible for delivering the email in case the user/admin releases it from the quarantine.

This is done via the "tblServersServerID" parameter in the SpamFilter.ini file. You will probably have that value set to 0:

tblServersServerID=0

That value is inserted for each email in the tblQuarantine table in the database in the column "ServerID".

If you configure your 1st SpamFilter to have "tblServersServerID=1" and your second SpamFilter to have "tblServersServerID=2", then each of the two SpamFilters will be delivering the emalls from the database that it originally archived if the user chooses to deliver them.

Here's what you can do however. Create an AFTER INSERT trigger on the tblQuarantine table which immediately resets the value of the "ServerID" to "2". This will force all quarantined emails to be handled by your second SpamFilter, which will then deliver them to your destination SMTP server.

Now for the existing emails in the database that have a "0" for the ServerID. That "0" is sort of a wildcard, and all SpamFilters are hardcoded to try delivering emails that have a null or a 0 in that field, so the first SpamFilter server that gets to that record will be the one to deliver that email if it's released from the database. You can of course run a query that changes all the "0" to "2" when you are ready to go live so all pre-existing emails will be handled by your 2nd SpamFilter.

Please let me know if you need assistance with the trigger.

-------------
Roberto Franceschetti
LogSat Software