Richard,

Removing the HTML tags requires little code but it does add more overhead we feel comfortable with right now.

As far as DNA fingerprinting, it is self-adjusting. If spammers decide to spell v1agra or v*i*a*g*r*a, after receiving a few such emails at first, the statistical engine will begin to recognize the new patterns and readjust itself. At least it is in our preliminary tests...

Roberto F.
LogSat Software

I don't believe that removing HTML tags prior to keyword searching would amount to a lot of code or overhead.

"DNA fingerprinting" has failed elsewhere because spammers are on to it, and throw random garbage into messages to change the "fingerprint" -- I hope your algorithm is smarter than others.

Another spammer trick alluded to earlier in this thread is the creative use of "alphabet substitutions" - such as 1 or ! or | for I or l , 0 for O, @ for a, etc., and peculiar spacing, with and without separating characters (-_+=~. are common).

If viagra is the keyword, then my wishlist is for the software to, in addition to stripping out HTML tags, match V!AGRA and Viagr@ and ###VIAGRA### (embedded in other text), V I A G R A, V-I-A-G-R-A  and so on.

It is interesting to note that (at least I think this is the case) HTML does not allow SRC= or HREF= to be broken up with comments - thus even if VIAGRA is obscured, the website mentioned within is not - thus in the HTML gobbeldygook example a few messages back - this is buried in the otherwise unreadable html: <A href="http://www.pure-herbal.biz/sknoc/vp/"

One should be able to filter out mail containing "pure-herbal.biz" -- if nothing else!

Having read this thread, I think the feature request that is coming out is a function that would render HTML embedded into a subject line back as text, and then accepting/rejecting the message based on the rendered subject line.  So "VIAGRA" and "VIA<crap>GRA" would all be caught if "VIAGRA" is in the keyword filter.

The question to LogSat then becomes, is this feature request feasible and if so, is it sensible?

Allan,

We had given thought to your same observations. There is a (big) performance penalty in having to parse incoming email's HTML, so we decided to keep it simple and just work with the email source.

We're working hard to prepare a new version that does DNA fingerprinting on incoming emails, which will greatly diminish the need to specify multiple keywords. Once this is complete, should it not have the desired accuracy, we may go back to implement HTML parsing as well.

Roberto F.
LogSat Software

==========

A new message by Allan Poulsen was posted in Support Forum

Re: Keyword filter

I do not quite agree with you. I have added some words in the keyword filter - regardles of their meaning, I want to filter the mails.

>

If Outlook ( and other mail programs ) is capable to filter out worthless HTML tags when displaying the message, I still cant see why the program can't do the same...

Greetings

Allan

Valid points, all...

Dan, your RegEx list is always welcome for reading and comparison.  You've said in a previous post that you're not a "RegEx expert", but it seems more and more that you're becoming one as a result of this project!  Filtering on a COMMON single word might not be such a good idea, but there are plenty of single-word spam traps that are very effective with zero false positives.  You'd be surprised how much spam I'm catching by filtering on the slang word "milf".  Both "vallum" and "vaiium" (consider the spammer's creative use of capitalization on the i's and l's) are effective at trapping a few dozen spam messages each week.

But back to my original concern... trapping the embed<crap>ded tags.  Unfortunately I don't think that even your most recent RegEx list addresses this scenario.  Thus, Allan's post about pre-filtering the HTML tags and then running keyword check does have merit.  A sentence that contains the phrase "pe<blurb>nis enl<!--okay-->argem<extract>ent" then becomes "penis enlargement" and it becomes a policy decision for the mail system's administrator whether or not to filter on those two words.

Still, this doesn't rate an enhancement request, not right now.  The statistical scoring enhancement that Roberto has mentioned in previous posts could have a more far-reaching impact on spam control than all of these blacklist/whitelist tweaks combined.  So, I'll just sit tight and wait for THAT beta to arrive.

There is one trick you can use to view the source in Outlook.  I use this on most of my mail before opening when it has gotten through SpamFilter but sill looks suspect.

1. disable any preview panes.  You should do this anyway because previewing an HTML email can cause an image to be downloaded, in which your IP (and possibly a unique code) will be logged and show your address as legitimate, causing more spam to come your way.

2. Highlight the message but do not open it.

3. Under "file" choose "save as ..."  If the message is a plain text email, it will add the .txt extension.  (Text emails will be safe to open in outlook).  HTML emails will be given the .htm extension.  I usually just save these to my desktop and then view the source in a HTML editor such as HomeSite.  Notepad will suffice.

I am new to regex and was wanting to know if you could you give me a break down on what a few of your lines are doing? namely lines 1, 2, 5 and 13.

Thanks in advance.

I do not quite agree with you. I have added some words in the keyword filter - regardles of their meaning, I want to filter the mails.

If Outlook ( and other mail programs ) is capable to filter out worthless HTML tags when displaying the message, I still cant see why the program can't do the same...

Greetings

Allan

A suggestion:

If Outlook can "filter" the unwanted HTML syntax and display the text, why can't SpamFilter when it's checking for keywords ?