Print Page | Close Window

RegEx Versus KeyWords

Printed From: LogSat Software
Category: Spam Filter ISP
Forum Name: Spam Filter ISP Support
Forum Description: General support for Spam Filter ISP
URL: https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=1244
Printed Date: 05 February 2025 at 3:28pm


Topic: RegEx Versus KeyWords
Posted By: Desperado
Subject: RegEx Versus KeyWords
Date Posted: 04 July 2003 at 2:43am

All, (esp George)

My understanding is that a few of you have very extensive Keyword lists.  I am using Regular Expressions ONLY and at this point only have 6 entries.  I was wondering if someone who is using a very large keyword list could give me a comparison of percent blocked by keyword (comparing to my stats below).  Also, I do not seem to be getting ANY False positives in my own mail and none of my customers seem to be complaining about false positives ... how does that compare with the long keyword list.  Any information will be very useful to me. Please see my current Quarantine Stats "Snap Shot" below.

Current Database Contents

Current Msg. total: 367654 Marked for deletion: 0
No Reverse DNS: 119583 IP in MAPS B/L: 150683
Email From B/L: 167 Exc'd max RCPT TO: 927
Banned Keywords: 96123 Email TO B/L: 4
Domain B/L: 167 IP B/L: 0
Blocked by Bluto: 320028 Blocked by Raptor: 47626

Primary SMTP Statistics   (From May 25 2003)

Total Inbound Conn.: 1391086 EMail Attempts: 1393792
EMails Forwarded: 436543 EMails Blocked: 1291733

Dan S.


Replies:
Posted By: Guests
Date Posted: 04 July 2003 at 9:30am

I see you have no-reverse-dns turned on... we get hundreds of false positives from that feature and had to disable it... Are you not seeing the same kind of problem ???

 

Roy

Posted By: Desperado
Date Posted: 04 July 2003 at 9:45am

We do not consider No EDNS to be "False Positive". We consider that to be a configuration problem on the senders side.  They either fix it, or we do not receive their mail. We have never accepted connections with no RDNS even before SpamFilter ISP so our customers are used to having the sender contact their carrier to get it fixed.  We have a very clear policy on that and soon, ALL US ISP's will be doing the same (we hope).

There are always cases where the sender's carrier are jerks or laze but we usually shame them into fixing their DNS.

Dan S.

Posted By: Desperado
Date Posted: 04 July 2003 at 10:03am

SORRY  NO "RDNS" ... I am very tired!

Dan

Posted By: dcook
Date Posted: 05 July 2003 at 9:17am

Hi Dan,

Could you share your regex list.  I am still gathering blocking tips.  


Posted By: Desperado
Date Posted: 05 July 2003 at 12:01pm

Dwight,

 

I answered this in detail and then got a "script Timeout" when posting.  (ARRRRRGGGG!) I will try to generate the text locally and paste it in.


Posted By: Desperado
Date Posted: 05 July 2003 at 12:39pm
Dwight,
 
Now I am lazy ... here is a "not so detailed version of what I wrote the first time.
 
I am still working on my VERY short list and am also working on the order of things in my other lists.  When you look at my "Stats", you will see that relays.osirusoft.com has very few "hits" That is because I killed off all the messages that had been blocked by that list so I could see if it is even worth having now that I changed the order.  relays.osirusoft.com is a very good list but is slow to respond at times and other lists carry the same blocks.
 
Back on the topic:
 
(<[!--]+[a-zA-Z0-9]{11,})     >>>>> Roberto's (modified) "magic" html comment blocker

( http:///w{0,6}%[/d" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - http://" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - http:// \w{0,6}%[\d])     >>>>> Blocks href's to partially obscured URL's.  WARNING!  Until I work something out with my buddy at "PayPal"  (he is one of their developers), I had to add mailto:*@paypal.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@paypal.com to my allowed FROM list.

(href=" http://+[/d" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - http://+" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - http://+ [\d])     >>>>> Blocks any href to "Dotted IP's"

(content-type: text/html\r\ncontent-transfer-encoding: base64)   >>>>>>  This one is supposed to block Base 64 encoded HTML that isn't tagged as an inline image.  It seems to work sometimes and not others.  I have not found the difference between the ones blocked and the ones not blocked yet.
 
The "Clear Text" keywords below are only in until I find the "Correct" pattern to us a RegEx.

Mobutu Sese Seko
banhi.com
text-decoration: blink
 
I am working on an SQL query that will run once every 24 hours to give me a breakdown on how many of each expression blocked in the 24 hour period.  This will help me determine if a keyword is even worth having in or not.
 
Here are my latest stats
 

Current Database Contents

Current Msg. total: 362076 Marked for deletion: 0
No Reverse DNS: 117088 IP in MAPS B/L: 139252
Email From B/L: 254 Exc'd max RCPT TO: 1916
Banned Keywords: 103535 Email TO B/L: 25
Domain B/L: 7 IP B/L: 0
Blocked by Bluto: 315470 Blocked by Raptor: 46607

Primary SMTP Statistics   (Last Cleared May 25 2003)

Total Inbound Conn.: 1421750 EMail Attempts: 1423184
EMails Forwarded: 443379 EMails Blocked: 1327058

Secondary SMTP Statistics   (Last Clear


Posted By: dcook
Date Posted: 06 July 2003 at 3:48pm

Thanks Dan!

You have some good ideas.  I have put some into practice. Are you getting many false positives for dotted ip addresses?

I also wanted to ask if you could provide a link to your adminlogin.asp page for the stats.  That page is missing from your stats download post.

Dwight

 

 

Posted By: Guests
Date Posted: 06 July 2003 at 8:02pm

Dwight,

Second question first ... The stats were a copy from our administrator page so I can't really give access to the actual web site.  Only our admins have that.  What stats were you looking for?

First question:  I guess we have to define "False Positive" for that one.  As an ISP, we have taken the stance that there is no excuse for "dotted IP's" in a URL.  If you can't put a DNS host name in, and validate some traceable ownership of that IP, we view that as obfuscation. By our definition, that's Spam.  So, by our definition, we have no false positives.

We have had cases where perfectly valid email had dotted IP's in a url and our customer informed their sending party and they fixed it.

I am not sure that was what you wanted to hear but it is our policy.  Again, what stats are you looking for?  I can try to give you a "snapshot" of our stats but I can't give public access to the site itseld.

Dan S.

Posted By: Guests
Date Posted: 09 July 2003 at 12:17pm
Is that a snapshot of the asp web pages provided by logsat? I haven't attempted to set it up yet but thats looks very cool.

Posted By: Desperado
Date Posted: 09 July 2003 at 12:30pm

No.  I have written a complete application in ASP for my customere, my "advanced" customers, and our internal Administrators.  The snapshot came from my advanced manager.  I see that the snapshot was "cut off" at the bottom.  I may put a link up of some sample screen shots to give users some ideas on what can be easily done to get some of this information out of the system.

Dan S.

Posted By: Guests
Date Posted: 16 July 2003 at 2:53pm

Dan,

Reading your stats, what is "Bluto"? Also, presume Raptor is the firewall.

John

Posted By: Desperado
Date Posted: 16 July 2003 at 4:56pm

John,

Actually, we have a primary and a secondary mail server.  The primary is "Bluto" and the secondary is "Raptor".  They are located in 2 separate locations but "report" to the same MS SQL server.

Dan S.

 

Posted By: Guests
Date Posted: 30 August 2003 at 2:15am

Hello,

Could you please post a list of the regular expressions that you use so that I may implement them here? Any help would be greatly appreciated.

Dean

Posted By: Desperado
Date Posted: 30 August 2003 at 2:34am

Dean,

The list I CURRENTLY use is as below HOWEVER ..... my list blocks a bunch of "Lists" that use a lot of extra garbage in their messages so I also have included a list of "Excluded From Addresses"

KEYWORDS:

((http|3dhttp)://.{0,26}(((%.+%))|@|:)[(\d|\w)])
( http://+[\d]{1,3}\.{1}[\d]{1,3}\.{1}[\d]{1,3}\.{1}[\d]{1,3" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - http://+" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - http://+ [\d]{1,3}\.{1}[\d]{1,3}\.{1}[\d]{1,3}\.{1}[\d]{1,3})
((<[!--]+[\x20]{0,1}[a-zA-Z0-9]{10,}[\x20]{0,1}[!--](.+)){2,})
(<[!--]+[a-zA-Z0-9]{2}(-->))
(( http://http:/" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - http://http:/ \w)|(<(\w){3,10}(\x20/>)|(\* http://w" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - http://w )))
(<(!-- )+[a-zA-Z0-9=]{28,}( -->))
(content\-type:\x20text/(html|plain)(;{0,1})\r\ncontent-transfer\-encoding:\x20base64\r\n)
((limited time (special|offer)))
(((arge your p)|(3 - 5 inches\!)|(herbalpillsonline)|(herbaltrials\.com)|(pillsavings)|(gsc\-100)))
((text\-decoration: blink)|(click here to start))
((your privacy is extremely important to us)|(this is not spam))
( http://www.(\w){1,20}(4u).(biz|com|net" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - http://www." CLASS="ASPForums" TITLE="WARNING: URL created by poster. - http://www. (\w){1,20}(4u).(biz|com|net))
((application\.zip|details\.zip|document_9446\.zip|document_all\.zip|movie0045\.zip|thank_you\.zip|your_details\.zip|your_document\.zip|wicked_scr\.zip))
((re: )+(wicked screensaver|details|approved|thank you!|that movie|your application|re: my details))
(See the attached file for details)
(this email address will be expiring)
selectgroupmedia.com
getit4less.biz
generic viagra
Unsubscribe By Postal
debtelimination


Excluded From Addresses:

mailto:*@paypal.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@paypal.com
mailto:*@listproc.pcworld.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@listproc.pcworld.com
mailto:*@industryweek.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@industryweek.com
mailto:*@gpsadvantage.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@gpsadvantage.com
mailto:*@gwbakeries.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@gwbakeries.com
mailto:*@peoples.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@peoples.com
mailto:*@*.lga2.nytimes.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@*.lga2.nytimes.com
mailto:*@*.*.nytimes.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@*.*.nytimes.com
mailto:*@softshare.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@softshare.com
mailto:*@regulusgroup.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@regulusgroup.com
mailto:*.*@dell.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *.*@dell.com
mailto:*@e-news.fsonline.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@e-news.fsonline.com
mailto:*@lists.n-email.net" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@lists.n-email.net
mailto:*@lists.techtarget.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@lists.techtarget.com
mailto:*@lyris.stockupticks.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@lyris.stockupticks.com
mailto:*@multexinvestornetwork.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@multexinvestornetwork.com
mailto:*@newsletter.online.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@newsletter.online.com
mailto:*@insightmedia.info" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@insightmedia.info
mailto:*@nhfairfield.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@nhfairfield.com
mailto:*@newhorizons.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@newhorizons.com
mailto:*@rootsweb.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@rootsweb.com
mailto:*@*.rootsweb.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@*.rootsweb.com
mailto:*@returns.groups.yahoo.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@returns.groups.yahoo.com
mailto:*@cygnuspub.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@cygnuspub.com
mailto:*@*.classmates.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@*.classmates.com
mailto:*@listserv.usairways.com" CLASS="ASPForums" TITLE="WARNING: URL created by poster. - *@listserv.usairways.com

Regards,

Dan S.

Posted By: Guests
Date Posted: 02 September 2003 at 2:11pm

Dan,

Getting some false positives with this latest post.  I previously was using your regex from about a month ago, which I think was producing fewer false positives, yet letting more spam in.  Any ideas?

--Clator


Print Page | Close Window