Hi,

We see loads of email munged with html tags like this:

V<!fjdskfdshjk>ia<!jfdkjfdskfdfjjjjjj>gra!

I have some RegExes working well to block these, but I worry about false positives because they don't necessarily have to match the whole keyword:

I agree that looking for the munge should be a more effective tactic than un-munging and looking for keywords.  Unfortunately certain types of legitimate email (like those containing XML content) also have a whole lot of comment tags embedded in them, so false positives become a problem.

To solve this, a generic prefiltering mechanism could be applied in a different way than I described before.  I suggested doing this:

((~r)<.*>) --> ((?i)viagra)

where "(~r)" was a modifier to remove text matching the following RegEx, "<.*>", and "-->" was an operator to pipe the output into the next filter RegEx, "((?i)viagra)".  We could have other operators like "!->", meaning only pipe the output to the next filter if the first filter returned false.  Then we could construct something like this:

(.*XML.*) !-> (([^>]<!.*>.*[^ $]){3,})

where the first RegEx looks for an XML identifier and only if it doesn't find one the message text is passed to the second filter, which looks for 3 comment tags on a single line.  (That filter is a bit buggy regarding newline characters, BTW).

It would take the spammers a little while to figure out that they just need to stick an XML identifier into their munged spam to get it through...

Alec,

Applying DNA fingerprinting to detect spam is proving very effective in detecting all tricks used by spammers. Our statistical engine is able to cope with junk inserted a-la XML content and/or html comments. All meaningless words and random strings of characters are simply ignored by the filter. Almost all messages do have some DNA structure in them will allow SpamFilter to consider them spam or not.

We have released a beta version with this new anti-spam engine if you wish to test-drive it.

Roberto F.
LogSat Software

From our beta download page at http://www.logsat.com/sfi-beta.asp:

Roberto F.
LogSat Software

======================

The new v2.x release of SpamFilter will feature a statistical DNA fingerprinting of incoming emails. The statistical analysis is performed using Bayesian rules. Tokens within incoming emails are scanned and catagorized in a corpus file. The content of all new incoming email is fingerprinted and checked against the historical data. If there is a high statistical probability that the email is spam, it is rejected.

The statistical engine currently kicks in after 500 non-spam and 500 spam emails have been received. This is done to build a valid statistical base to use before emails are rejected. During this period of time, it is critical to avoid false positives. If a good email is quarantined, forcing it's redelivery either thru the web interface or the SpamFilter GUI will "teach" SpamFilter that the fingerprint in that email is a "good" one, and the statistical DNA database will adapt itself to it. It is very important initially to check the quarantine often to force delivery of legitimate email that has been blocked by the "regular" filtering rules.
==============================