Problem with Bad Recipients |
Post Reply |
Author | |
ecarbone
Newbie Joined: 11 January 2014 Location: Mexico Status: Offline Points: 24 |
Post Options
Thanks(0)
Posted: 30 October 2015 at 12:48am |
Hello, I've found an strange behavior with SpamFilter, Exchange Server 2013 and Exchange recipient filter feature enabled. When a user sends an email and he types a bad address along with a good address, the message is NOT delivered to the good address and Spamfilter logs a "User unknown error" and tries to send a NDR to Sender. For example, lets suppose I send an email like: From: Someone@gmail.com To: validuser@domain.com, InvalidUser@domain.com Subject: Whatever... Tracing LOG files at SpamFilter I can see that message is correctly sent to the Exchange Server at domain.com after all the internal spam checks (SPF, Bayesian,SFDB, etc, etc) that is "Created thread (XXXXXX) to handle delivery" From Exchange Server side I can see that message arrives correctly and Exchange detects that one of the recipients is invalid because it doesn't exist at the company. Exchange sends back a Recipient OK for validuser@domain.com and also sends back a User Unknown for InvalidUser@domain.com It seems that SpamFilter is just processing the "User unknown" response and therefore rejects the message without trying to send it for the valid user. Hope you can help, I can provide LOGS from SpamFilter side and Exchange side if needed. By the way, I'm aware that SpamFilter has an "Authorized TO Emails" White List; but by the moment we are not able to Sync this list with our customers Address Books in an efficient way, so they rely on Exchange's recipient filter to reject invalid recipients. Also worth to note that Exchange 2013 has FrontEnd/BackEnd architecture, that is .. all messages are received by a FrontEnd Server on port 25 with standard SMTP, after analyzing the destination recipients, messages are routed to a BackEnd server depending on which server the mailbox resides; this is valid even if there is only 1 server at the company. Exchange side is NOT USING Edge roles, our customers are seeing SpamFilter as the first line of defense for cleaning spam. Regards, Enrique.
|
|
Enrique
|
|
LogSat
Admin Group Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
Post Options
Thanks(0)
|
Enrique, If SpamFilter has this entry in the SpamFilter.ini file (it's enabled by default): VerifyRCPTTOForCleanEmails=1 Then it should be able to detect when your Exchange server rejects a recipient. When this happens, SpamFilter will by design reject the email with a 5xx error. This is done on purpose as behaving this way forces the remote server that is sending the email to send a bounce NDR (non-deliverable receipt) email back to the sender to notify them that the email was not delivered correctly. If this didn't happen (meaning that SpamFilter accepted the email), then it would be up to SpamFilter to send an NDR back to the sender. As often times spammers will send email using fake or spoofed email addresses, this would be bad as SpamFilter would then be sending large amount of backscatter emails out to innocent victims and may cause your server to be blacklisted. Note that in either case (whether the sender's own mail server or SpamFilter) sends the NDR, the sender will get a bounce email back saying that their email was not delivered to all recipients. So it may very well be that we leave this job for the remote sending server...! In the rejection message, SpamFilter will include the rejection message with which your Exchange server has rejected the recipient. The same error message will be logged in SpamFilter's logfile. Please see the following example below where we send an email to SpamFilter containing both an Valid and an Invalid email address. As you can see, at the end of the transaction SpamFilter will issue the 550 error reporting back to the sender the error that our own mail server generated (550 Requested action not taken: mailbox unavailable or not local ). It will be up to the sender to then send a bounce back to the sender (roberto@test.logsat.com) to let them know that the email was not delivered to "Invalid@logsat.com". SpamFilter logfile: 10/30/15 15:03:32:674 -- (44285680) EMail from roberto@test.logsat.com to Valid@logsat.com, Invalid@logsat.com --- was forwarded to mail2.netwide.net:587 - Response:250 Requested mail action okay, completed --- 10/30/15 15:03:32:674 -- (44285680) Some recipients do not exist, sending 550 to sender SMTP server. The destination SMTP server said:Invalid@logsat.com: 550 Requested action not taken: mailbox unavailable or not local --- SMTP session trace: 220 test.logsat.com Welcome to SpamFilterISP SMTP Server v4.7.1.172 EHLO cmctrf3 250-Hello cmctrf3 250-AUTH LOGIN 250-ENHANCEDSTATUSCODES 250-SIZE 25600000 250-8BITMIME 250 BINARYMIME MAIL FROM:<roberto@test.logsat.com> 250 2.1.0 roberto@test.logsat.com Address Okay RCPT TO:<Valid@logsat.com> 250 2.1.5 Valid@logsat.com Address Okay RCPT TO:<Invalid@logsat.com> 250 2.1.5 Invalid@logsat.com Address Okay DATA 354 Start mail input; end with <CRLF>.<CRLF> From: roberto@test.logsat.com To: Valid@logsat.com CC: Invalid@logsat.com Subject: test 1 Date: Fri, 30 Oct 2015 00:48:40 -0400 test . QUIT 550 --- The email was not delivered to the following addresses: Invalid@logsat.com: 550 Requested action not taken: mailbox unavailable or not local ## Edited by LogSat - 30 October 2015 at 3:34pm |
|
ecarbone
Newbie Joined: 11 January 2014 Location: Mexico Status: Offline Points: 24 |
Post Options
Thanks(0)
|
Roberto,
Thanks for your reply and I do understand what you are saying. In your simple Invalid@logsat.com is part of the "CC" of the message and Valid@logsat.com is part of the "TO" of the message. The behaviour I'm watching is that when both recipients are part of the "TO" section of the message and Exchange's recipent filter is enabled, in that case, Exchange sents back 1 RCPT OK and 1 InvalidUser message back to SpamFilter, but Spamfilters at that moment rejects the whole message. I made a test turning off Exchange Recipient Filtering and SPamfilter receives a RCPT OK and in turn forwards the message, later in the pipeline Exchange detects that 1 of the recipients is invalid and sends the NDR to the Sender. Hope I explained myself, the problem shows when Invalid and Valid are part of the "TO" Header at the same time, I have not tested putting both recipients in the "CC" or both at the "BCC", but I will do the test and report back. Regards, Enrique |
|
Enrique
|
|
LogSat
Admin Group Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
Post Options
Thanks(0)
|
Enrique,
A mail server only knows of a recipient when the sender's SMTP server has issued a "RCPT TO" command during the SMTP session. The mail server does not look inside an email's content to see who the recipients are. Thus the various "To:", "CC:" and "BCC" headers are meaningless to a mail server as they are only part of the content of the email. By the time the content of the email is sent after the DATA command, the sender and the recipients have already been determined via the MAIL FROM and RCPT TO commands. So in my example, the only two addresses that matter are these two: RCPT TO:<Valid@logsat.com> RCPT TO:<Invalid@logsat.com> The mail server (both SpamFilter and our destination SMTP server) do not know and do not care if these emails were meant to be part of the TO, CC or BCC addresses - it is irrelevant. All SpamFilter and our mail server need to know is that they need to deliver the content of the email (after the DATA command) to those email addresses. Again - SpamFilter and our mail server do not look inside the body of the email to see what the TO and CC email addresses are and how they compare to the RCPT TO addresses (actually the latest versions SpamFilter will log if they are different, but only to give admins more options to find emails in logs...). Going back to what you see by turning on and then off the Exchange Recipient Filtering - when you turn it on, your Exchange server will reject the invalid recipient when SpamFilter attempts to forward it the email with the non-existant email address. When this happens, SpamFilter will (correctly) reject the email from the sender by outputting a 5xx error code and in the bounce message will include the rejection notice from your Exchange server. When you disabled the Recipient Filtering, your Exchange server will accept the email from SpamFilter even though there is an invalid recipient. SpamFilter will see it delivered successfully so as far as SpamFilter is concerned it was delivered. Your Exchange server then, after receiving the email, will see the invalid recipient and will then in turn send a bounce email back to them. Please note that this is an actual email being sent by your Exchange server. It will be sent to whatever email address the sender has specified in their "MAIL FROM" command, which in my example above is: MAIL FROM:<roberto@test.logsat.com> Note that while in most legitimate emails sent by humans this email address will match the address specified in the "From:" header in an email. However in most emails sent by spammers this will be a spoofed email address belonging often times to an unaware persona/domain, and in case of automated emails, the MAIL FROM email address usually mis-matches the one specified in the "From:" header. Due to this, with the Recipient Filtering disabled in Exchange, you run in the high risk of having your Exchange server sending large amounts of NDR outbound email in response to spammers and/or automated emails that will be sent to innocent or invalid email addresses. The recommended way to proceed is to leave the Recipient Filtering enabled, and allowing SpamFilter to reject the email with the incorrect recipient(s). When the sender sends an email with one or more invalid recipients, that sender will always receive a non-deliverable email bounce back. You can either (1) let that happen by having their own ISP send them the bounce NDR (by having SpamFilter reject the email), or (2) by having the NDR email being sent by your own server either by disabling Exchange's Recipient Filtering or by changing SpamFilter's default value for VerifyRCPTTOForCleanEmails=1. It's usually better to avoid backscatter originating form your network, saving network resources, and avoiding being blacklisted by going with option 1. PS - as a side note - the BCC header only appears in the email residing in the sender's own mailbox - the BCC header is stripped when the email client actually sends the email to the mail server
|
|
ecarbone
Newbie Joined: 11 January 2014 Location: Mexico Status: Offline Points: 24 |
Post Options
Thanks(0)
|
Roberto,
Thanks again for your help. The strange behavior is that email is not arriving to the valid recipient, the whole message is rejected. I understand your explanation about TO,CC,BCC but curiously if I place valid and invalid on the same TO, message doesn't arrive to valid. If I place valid and invalid, one in TO and one in CC, message arrives to valid as expected. Regards, Enrique |
|
Enrique
|
|
LogSat
Admin Group Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
Post Options
Thanks(0)
|
That would be odd indeed if the valid user is not receiving the email. Can you please let me know what version of SpamFilter you are using, and then provide us with the following:
If the zipped file is over 8MB in size, I'll send you via a PM the upload details when you can send us the files (or we can pick them up from a location you can provide) |
|
ecarbone
Newbie Joined: 11 January 2014 Location: Mexico Status: Offline Points: 24 |
Post Options
Thanks(0)
|
Thanks a Lot, I'll prepare the info and be back.
|
|
Enrique
|
|
ecarbone
Newbie Joined: 11 January 2014 Location: Mexico Status: Offline Points: 24 |
Post Options
Thanks(0)
|
Sorry for the Delay, I'm attaching the requested info using the repository sent by email.
Regards Enrique |
|
Enrique
|
|
LogSat
Admin Group Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
Post Options
Thanks(0)
|
Enrique, From your logs I believe what you call the front end server is where SpamFilter forwards its emails to. If that is the case, the front end Exchange server is the one that is rejecting the entire message if one of the recipients is incorrect. Furthermore, it does not reject the recipient right away when the RCPT TO command is issued, but rather only at the end of the SMTP session, when the email has been transmitted. To make matters worse, the front end server is not specifying which email address is incorrect - it simply rejects the entire message. Please note that from your logs I can confirm what I mentioned in the forum - the behavior does not change depending on whether the invalid email address is specified in the CC or in the TO headers - the SMTP session is the same and the front end server rejects it in the same way. These are the transcripts of what happens to an email sent to eca...@...com.mx and invalid@......com.mx to your front end server at ...........com.mx. The first session is with invalid@.....com.mx entered in the TO field, the second session with invalid@.....com.mx entered as a CC. As you can see, your front end server accepts both recipients without any errors, and at the end it rejects the entire message with a "550 5.1.1 User unknown" without specifying what is the incorrect user: 220 ........com.mx Microsoft ESMTP MAIL Service ready at Mon, 2 Nov 2015 19:43:36 -0600 EHLO cmct...... 250-k..............i.com.mx Hello [] 250-SIZE 10000000 250-AUTH NTLM 250-8BITMIME 250 OK MAIL FROM:<roberto@test.logsat.com> 250 2.1.0 Sender OK RCPT TO:<eeca...@...com.mx> 250 2.1.5 Recipient OK RCPT TO:<invalid@.....com.mx> 250 2.1.5 Recipient OK DATA 354 Start mail input; end with <CRLF>.<CRLF> From: roberto@test.logsat.com To: eca...@...com.mx, invalid@.....com.mx Subject: Test with 2 TOs from LogSat - please ignore Date: Mon, 2 Nov 2015 18:50:39 -0600 Content-Type: text/html;charset="ISO-8859-1"; test please ignore . QUIT 550 5.1.1 User unknown 220 .........com.mx Microsoft ESMTP MAIL Service ready at Mon, 2 Nov 2015 19:38:07 -0600 EHLO cm.... 250-......om.mx Hello [] 250-SIZE 10000000 250-AUTH NTLM 250-8BITMIME 250 OK MAIL FROM:<roberto@test.logsat.com> 250 2.1.0 Sender OK RCPT TO:<eca...@...com.mx> 250 2.1.5 Recipient OK RCPT TO:<invalid@.....com.mx> 250 2.1.5 Recipient OK DATA 354 Start mail input; end with <CRLF>.<CRLF> From: roberto@test.logsat.com To: eca...@...com.mx CC: invalid@.....com.mx Subject: Test from LogSat - please ignore Date: Mon, 2 Nov 2015 18:48:39 -0600 Content-Type: text/html;charset="ISO-8859-1"; test please ignore . QUIT 550 5.1.1 User unknown As this error occurs *after* the recipients are send and accepted, SpamFilter must (per RFC) assume that the email is not delivered, and at this point SpamFilter does need to send an NDR bounce email back to the sender. Here is another problem however - SpamFilter never sends an NDR directly out to the internet, but rather it forwards the NDR to your destination SMTP server for delivery. In this case, the NDR is send to ..........com.mx, which however does not allow SpamFilter to open relay, so it does not deliver the NDR. You must ensure that your front end server whitelists SpamFilter so that the NDRs can be sent. Please note that if your front end server had rejected a recipient during the RCPT TO command, then SpamFilter would have known it was an invalid user as I mentioned in the forum, and it would not need to send an NDR since it would instead reject the email specifying in the 550 error code which email address had the problem. Also please note that I see from your backend server logs that the front end server is forwarding the email to the backend server even though it's reporting the email as being rejected to SpamFilter. This is very confusing as it causes emails to be delivered even though the front end server is reporting to having rejected them as a whole. I hope this helps a bit in understanding how the email flows and that your front end server should be configured differently. To summarize, these are the problems with that front end server: 1. It must whitelist SpamFilter so that NDRs can be delivered 2. It should reject invalid email addresses when the RCPT TO command is used to specify a recipient, not at the end of the email. 3. If it cannot be configured as in #2 above, it must at least specify in the 550 error message which email address was not deliverable so this can be reported back to the sender. Otherwise the error will be interpreted as "your email was not delivered to ANY of the recipients". |
|
ecarbone
Newbie Joined: 11 January 2014 Location: Mexico Status: Offline Points: 24 |
Post Options
Thanks(0)
|
Hi Roberto,
I really appreciate all your help and efort with this case. I'll take a closer look at what you are telling me. With Exchange, mail flow is suppose to go thru a "Front-End" service, message is accepted in "full" and then internally transfered to a "Back-End" service, in that part of the pipe-line is when Exchange Recipient Filter kicks in, detecting that 1 of the recipients is invalid and notifying to the "Front-End" service which in turn, notifies to SpamFilter. Now, after your explanation, I do understand that once "Front-End" receives the message, SpamFilter's work is done, so the problema must reside between Exchange's Front-Back communication. I've modified my SpamFilter.Ini in order to send NDR to a relay-aware server with have and also added Spamfilter's IP in order to be able to "talk" to this server, with this; sender will be able to know his/her message was not delivered. I don't know if this can be done on a per-domain basis, it will be great if that could be done. Once again, thank you very muy for your help and very detailed explanation, that makes my commitment with your product a stronger one. Regards, Enrique |
|
Enrique
|
|
Post Reply | |
Tweet
|
Forum Jump | Forum Permissions You cannot post new topics in this forum You cannot reply to topics in this forum You cannot delete your posts in this forum You cannot edit your posts in this forum You cannot create polls in this forum You cannot vote in polls in this forum |
This page was generated in 0.195 seconds.