filtering from addresses with : in them
Printed From: LogSat Software
Category: Spam Filter ISP
Forum Name: Spam Filter ISP Support
Forum Description: General support for Spam Filter ISP
URL: https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=7104
Printed Date: 27 December 2024 at 1:56am
Topic: filtering from addresses with : in them
Posted By: Terry
Subject: filtering from addresses with : in them
Date Posted: 21 November 2014 at 4:51pm
|
our CIO is complaining about spam emails making it to his inbox with no from address displaying in outlook. When I get to the headers of the email I see it's real from address is: From: Critical: Lower Blood Sugar Fix < mailto:Elias_Gibson@bloodglucnet.com" rel="nofollow - Elias_Gibson@bloodglucnet.com >; could I safely block this type of message by adding a blacklist entry for the from emails of *:* I don't believe any email name or address should ligitimately have a : in them but any guidance on this would be appreciated...(of course it would have been nice if this had been caught as spam)
|
Replies:
Posted By: LogSat
Date Posted: 21 November 2014 at 6:16pm
|
Hi Terry, The address in the "From:" header is what a email client displays to the user. The "From:" header is used by email clients to display any email address (including a portion with a name and punctuation signs) the sender wants, it's not necessarily the real address were email bounces or replies are sent to. SpamFilter thus ignores any addresses specified in the "From" headers. The true email address that the email is sent from (at least as indicated by the remote SMTP server) is the one in the MAIL FROM command. It will appear in the email headers in the "Return-Path:" header. SpamFilter will also log it in the "X-SF-RX-Return-Path" header. This said, the blacklists will work on the true email address (the one in the Return-Path header), not in the often fakes "From:" header. It's thus not possible to use the emails blacklist to stop emails based not he email address in the "From" header. While by default SpamFilter does not scan for keywords in the email's headers, this behavior can be changed with this option in the SpamFilter.ini file: ;if ScanAllHeaders is set to 1 SpamFilter will add all email's headers to the text examined for keywords and statistical Bayesian searches. ScanAllHeaders=1 With that option enabled, you could construct a keyword that will look for any lines that begin with "From:" and have the undesired portions of the email address you wish to block. It's usually better to use a RegEx (Regular Expression) to construct this keyword entry for more flexibility. For example this RegEx: ((?i)^from:.*\w.+:) will search for, case-insensitively, for any line in the email that begins with the word "From:" followed by one or more words, followed by a colon ":" character. It would block the "From:" sample you provided above. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: yapadu
Date Posted: 20 December 2014 at 3:22am
|
Are the spamfilter searches single line? I've never had any luck with ^ working as a start of line when doing spamfilter regex.
------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: LogSat
Date Posted: 20 December 2014 at 4:02pm
|
With RegEx the entire body (the text portions and any html section as well if present) will be checked, so multiline RegEx expressions should work. What expression are you trying to use? While I'm not a RegEx expert, I could try to help... ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: yapadu
Date Posted: 21 December 2014 at 12:04am
|
I don't have a specific example at the moment. I've never been able to figure out how to engage multiline to be able to use ^ in an expression. I don't think the example you give above would work. Using the regex test window in spamfilter even something as simple as this would not work: ^from or this (^from) I use Rad Software Regular Expression Designer for testing my regex. I leave the singleline option enabled in that software as I find that matches what spamfilter's regex does - meaning the ^ would only match at the very start of the string - not each line. Is there any recent user guide available? I installed my spamfilter way back and just apply the updates so I don't have any recent manual. I would like to go over the regex section again and see what modifiers are available.... maybe I am missing something to make multiline work so I could use ^ ------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: yapadu
Date Posted: 21 December 2014 at 12:16am
|
I downloaded a recent installer from the website and extracted the documentation, will check the regex and see if I'm missing something and report back. ------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: yapadu
Date Posted: 21 December 2014 at 12:34am
|
OK, after a bit of documentation reading I could change your expression to this: ((?im)^from:.*\w.+:) and it works in the tester window. I have added it to my own keyword list and will see if it catches anything, even though we don't have a known spam problem that the OP stated. A couple of other things to keep in my with spamfilter regex, it can't be the first keyword. You must use a non reg keyword first, so something like: http,((?i)^from:.*\w.+:) Also you can't use any , in any expression. The manual makes mention of things like this being allowed 555{3,6} But that actually will not work because the , conflicts with the comma delimited list of keywords. ------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: LogSat
Date Posted: 21 December 2014 at 1:22pm
|
Ouch, you're right, there is a bug with the comma. If you only have a single RegEx expression for a keyword, for example you have this entry: (555{3,6}) then the comma within the RegEx is interpreted correctly - there are no issues. If however you have a combined keyword, where the first part is a regular non-RegEx keyword, followed by a RegEx expression, then I did duplicate the bug you mention - the comma within the RegEx is interpreted as a separator. So to make this clear, if this combined keyword is used: test,(555{3,6}) then SpamFilter will interpret this as 3 different keywords: test 555{3 6} going back to your original question about multiline searches, this is for example one of the keyword we ourselves are using to catch spam that contains nothing but the email I include below: (\s*[\n\r]*<div>[\n\r]*<a href.*</a>[\n\r]*</div>[\n\r]*\s*\Z) with: \s* - zero or more spaces [\n\r]* followed by zero or more CR or LF characters <div>[\n\r]* - followed by the <div> tag with zero or more CR LF characters after it <a href.*</a>[\n\r]* - then followed by an <a> <\a> tag followed by zero or more CR LF characters after it </div>[\n\r]*\s* \Z - The above expression MUST APPEAR AT THE END of the email ==================== Date: Sun, 21 Dec 2014 11:33:36 +0000 From: "Lola Cash" <lola_cash@technologictimes.com> Reply-To:"Lola Cash" <lola_cash@technologictimes.com> Message-ID: <56b22c0-58d6f-19@technologictimes.com> Subject: FW: Watch the 3d porno tube videos in HD resolution X-Priority: 3 (Normal) MIME-Version: 1.0 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: 8bit X-SF-RX-Return-Path: <lola_cash@technologictimes.com> X-SF-HELO-Domain: technologictimes.com X-SF-Originating-IP: 198.23.51.56 <div> <a href="http://americanfamilycamping.org/wp-content/plugins/jetpack/_inc/genericons/start.html">Watch the 3d porno tube videos in HD resolution</a> </div> ============================ ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: yapadu
Date Posted: 21 December 2014 at 8:28pm
|
I was under the impression that if you wanted to use regex, you had to use a non regex keyword first (don't know if that is true or not). Anyway, I tried this for a few hours: ((?im)^from:.*\w.+:) It stopped an amazing 20,000+ messages before I stopped it. It was triggering on stuff like this: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=facebookmail.com; s=s1024-2013-q3; t=1419179688; bh=zvKprC165Uj54NJsXq+FCPDpdJ5NmqBef3SlJ4isOW4=; h=Date:To:From:Subject:MIME-Version:Content-Type; b=fhb+3MdEvYOHHvdHJk42yTnMdRpcZz00VlVnYma3pYwTUEXhNCl9M7REYmT0inn93 wyUgbw3iYd9b+hFg09Jotcc6GdRpwdpseTL4oxcZHEWFCirpVGRXnTK7BML8Js9SWZ uUcZ45nDyTUXDtTa7pGvR5xjwzKDTpeWzybgn+J4= Received: from facebook.com (UdjrGX6wsOXSz0WD+9OTzGASp4gIIja7+HPf9pIIu7ymp0nNXCB4zGVb2G3cEBG+ 10.103.99.81) The question is, why? If the ^ means the start of a line, why is it triggering on that? ------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: LogSat
Date Posted: 22 December 2014 at 4:10pm
|
Uhm, are you sure that what you indicated is the line that triggers it? As you said, the "^" indicates the start of a line, and I think your RegEx would have stopped an entry like this as well (I bolded the part that would match): from: roberto franceschetti <roberto@test.logsat.com> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=facebookmail.com; s=s1024-2013-q3; t=1419179688; bh=zvKprC165Uj54NJsXq+FCPDpdJ5NmqBef3SlJ4isOW4=; h=Date:To:From:Subject:MIME-Version:Content-Type; ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: yapadu
Date Posted: 23 January 2015 at 7:56am
|
I'm not having any luck using the start of line character ^ I did get a stand alone regex to function though, I thought in order to use regex you had to use a regular keyword before you could use a regex. However, today I got this to trigger a few items: (http://[a-z]+.[a-z0-9]+.rocks) What I can't get to work correctly is the ^, like these do not seem to trigger anything. (^from:.*<.*\.click>) ((?im)^from:.*<.*\.click>) ------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: LogSat
Date Posted: 23 January 2015 at 10:21pm
|
yapadu, The RegEx engine in SpamFilter matches the "^" by default only at the very start of the string (the body of the email). If you want to search anywhere in the text you should use the "m" modifier to treat the text as multiline. From our RegEx documentation: m
So you would need to use your second expression to search: ((?im)^from:.*<.*\.click>) Are you certain of the remaining portion of the RegEx? For example, your keyword above would stop emails with the following text appearing at the beginning of any line in the body: from:test<test.com .click> It would not match either of these two lines however: mail from:test<test.com .click> or from:test<test.com click> The "\." indicates an escaped period, so "\.click" in RegEx means a match to ".click". Is this the expression you were looking for? ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: yapadu
Date Posted: 24 January 2015 at 2:01am
|
Yes, I want to stop .click. I think my expression options are saying multiline, not case sensitive. I want to stop mail that is from a .click email address in the from header line. Stuff like this: Message-ID: <472a92d26bb9f7517d517a4c882c2291@sistema.optdatacrm.click> From: "Claro Empresas" <emm@optdatacrm.click> Reply-To: emm@optdatacrm.click MIME-Version: 1.0 Using this expression (?im)^from:.*<.*\.click> I think it should stop the example above, but no joy. I don't know what I'm doing wrong, but it would be very cool if I could figure it out. ------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: LogSat
Date Posted: 24 January 2015 at 8:47am
|
Your second expression (the one in blue in my post) should work then. Do you have this option set in the SpamFilter.ini file (it defaults to 0): ;if ScanAllHeaders is set to 1 SpamFilter will add all email's headers to the text examined for keywords and statistical Bayesian searches. ScanAllHeaders=1 ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: yapadu
Date Posted: 24 January 2015 at 8:51am
|
Yes, I have header scanning enabled. It is possible that there really has not been a hit yet, the keyword scanning is quite far down on the list. Maybe I will add a few more using the ^ technique as I have never used it before successfully and see if something triggers it. ------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: LogSat
Date Posted: 24 January 2015 at 9:33am
|
FYI I sent you a few test emails containing that "from" header (Subject: test RegEx) to your yapadu email address, and you're right, they are not being stopped. I sent the same email to our own mail server, and it was successfully stopped by your keyword. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: yapadu
Date Posted: 24 January 2015 at 9:42am
|
Actually, I just checked my quarantine - and there are two messages there from you! Triggered by a .click rule, this one: Found Keywords: [((?im)^from:.*<.*\.click>)] ------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: LogSat
Date Posted: 24 January 2015 at 9:59am
|
I had received the 250 OK after sending the email, so it looked as they were delivered. Do you have the option to "send to null" (::NULL suffix) for the keyword? ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: yapadu
Date Posted: 24 January 2015 at 10:03am
No, or it would not have been in my quarantine  However, I do think we have some option enabled where if we 'accept' the message but the destination is quarantine then we do not send back a rejection notice... I think, that was some option in the .ini file but I have no idea what it was. When we report the message as rejected, the number of calls/email messages we get for support just go through the roof. ------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: yapadu
Date Posted: 24 January 2015 at 10:04am
|
I think it is this: ;Setting DoNotSendNDROnQuarantine to 1 will prevent generation of NDR when email are quarantined by causing SpamFilter *not* to send an error code when quarantining emails DoNotSendNDROnQuarantine=1 ------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: yapadu
Date Posted: 24 January 2015 at 11:20pm
|
Ok, I think I've got my problem sorted out. I ran these two: ((?im)^from:.*<.*\.rocks>) ((?im)^from:.*<.*\.click>) I have been surprised how little they have picked up (your two tests and a couple others). The 'others' it picked up were not what I wanted. It triggered on an email with a from: in the headers and an email ending with .rocks much further down (it was a bounce message). I want the from to be at the start of the line (which I get with the m option) but I do not want the scanning to span more than one line, so the entire match must be on the single line to prevent false positives. Playing around I found I need to disable the singleline option as well. So I currently have: (?im-s)^from:.*<.*\.rocks> (?im-s)^from:.*<.*\.click> I'm going to run those and see if I capture what I expect. ------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: yapadu
Date Posted: 04 February 2015 at 7:47pm
|
We have been able to get this format to work: (?im-s)^from:.*<.*\.rocks> It is designed to find a sender address in the from header that ends with .rocks - that works. We then try taking it a step further, say to any recipient at a .com address so we came up with this. ((?im-s)^from:.*<.*\.rocks>),((?im-s)^to:.*<?.*\.com>?) Individually these two regex would trigger, but we have not been able to see the combo of using both trigger. Is the problem related to the bug regarding to the , or should we be able to do multiple regex separated with a comma? ------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: LogSat
Date Posted: 04 February 2015 at 10:02pm
|
I'm at bit of a loss here. One part of the question is explainable. When SpamFilter see a "(" as the first character in a line containing keywords, it expects *everything* on that line to be a RegEx. So the comma "," you used trying to specify two separate RegEx expressions on the same line will not work as you intended. It won't tell SpamFilter to separate the two RegEx expressions, rather it is interpreted as part of the RegEx. So you would need to use a single RegEx expression to perform your search. Assuming the "From" header is specified before the "To" header in an email, the RegEx to use would thus be: ((?ims)^from:.*<.*\.rocks>.*^to:.*<?.*\.com>) and to cover the cases where the "From" header is specified after the "To" header, you would need to invert them: ((?ims)^to:.*<?.*\.coz>.*^from:.*<.*\.rockz>) The problem I have is that neither of those two keywords are working correctly. In my RegEx tests the first keyword above should stopped an email with these headers: Message-ID: <472a92d26bb9f7517d517a4c882c2291@sistema.optdatacrm.click> From: "Claro Empresas" <emm@optdatacrm.rocks> To: "LogSat" <test@logsat.com> Reply-To: emm@optdatacrm.click MIME-Version: 1.0 ...but when using that keyword within SpamFilter it is not triggering it. I can't find anything obviously wrong with the implementation of RegEx in SpamFilter, yet it's not catching it. I'll try looking into this again during the weekend. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: yapadu
Date Posted: 05 February 2015 at 12:29am
|
I have altered my regex, not like your examples but a technique I used previously with spamfitler to mix regular keywords with regex. I will report back if it works. ------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: yapadu
Date Posted: 05 February 2015 at 1:38am
|
AH! I did it. In real, not in testing I managed to get it working. Here is what I ended up with. from,((?im-s)^from:.*<.*\.rocks>),((?im-s)^to:.*<?.*\.com>?) So the first from is just a placeholder, basically every message should have that. Then I have two different regex that are separated with a comma. The good news is that works! ------------- -------------------------------------------------------------- I am a user of SF, not an employee. Use any advice offered at your own risk. |
Posted By: LogSat
Date Posted: 05 February 2015 at 5:08pm
|
E X C E L L E N T idea, thumbs up for coming up with it - a great workaround! ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |