Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - Blocking foreign character sets?
  FAQ FAQ  Forum Search   Register Register  Login Login

Blocking foreign character sets?

 Post Reply Post Reply
Author
Alan View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Alan Quote  Post ReplyReply Direct Link To This Post Topic: Blocking foreign character sets?
    Posted: 17 June 2004 at 4:39pm

It has been requested before that we be able to block various character sets.

I've notice a certain spammer when adds to the header:

 charset="iso-xxxx-0"  where xxxx is a random 4-digit number.

I assume this confuses SF into thinking it cannot read the "foreign" character set and passes it on through.  If this is the case, can we get a toggle to quarantine all messages that contain an unknown character set?

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 19 June 2004 at 10:05am

Alan,

SF is not confused by the non-standard charset, and continues to examine the content for keywords. However practically all the emails with invalid charsets are spam. While not a huge number, the more can be stopped the better. We're in the process of developing a new filter to block emails with invalid charsets, and are a week or two away to having a pre-release build with this option.

Roberto F.
LogSat Software

Back to Top
Alan View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Alan Quote  Post ReplyReply Direct Link To This Post Posted: 21 June 2004 at 1:03pm
Roberto, the spam emails with the oddball invalid character sets seem to keep getting passed through even though they contain keywords that would normally filter them.
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 21 June 2004 at 10:17pm

Alan,

SpamFilter should still read the text even though it's being tricked with invalid charsets. While we develop the new filter, could you send us a copy of such an email so we can examine it? Please ensure that you retrieve the original email headers and contents, as some email clients, like MS Outlook, will modify the original email content without letting the user know.

Roberto F.
LogSat Software

Back to Top
Alan View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Alan Quote  Post ReplyReply Direct Link To This Post Posted: 23 June 2004 at 11:16am

Unfortunately we use MS Outlook and all email that is passed through SF goes to Outlook, so you wouldn't be able to analyse.

However we are still getting spam getting through, apparantly using this loophole.  In some of the more recent ones I can spot three different filters that should have stopped the spam but did not.  I am convinced the "charset="iso-xxxx-x"" isssue is the problem and that it is preventing SF from doing it's job.  Even the built in "Mail From = Mail To" is not stopping them, as I believe SF doesn't think it can read the header.

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 24 June 2004 at 12:52am

Alan,

If you're not able to see the original source of the email, please note that it's very possible that the source is formatted in a very different way than what you're seing in Outlook, and the keywords may not be working for that reason, not because of the incorrect charset.

Roberto F.
LogSat Software

Back to Top
Alan View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Alan Quote  Post ReplyReply Direct Link To This Post Posted: 25 June 2004 at 3:24pm

You say that SF DOES scan the contents and it is not being tricked by the fake character set.

It it is being scanned, then why does the spam get through when it contains several triggers that my filters would normally have caught?

Back to Top
Matt View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Matt Quote  Post ReplyReply Direct Link To This Post Posted: 26 June 2004 at 5:36pm

First try this freeware program to get the full headers using Outlook. They can be emailed back to you or another email address or sent to the clipboard so you can paste into an email or text file/document:

ftp://ftp.idp.net/AntiSpamTools/spamsource21_free.exe

Then restart your SpamFilter service and see what your results look like when you can verify your source code.  Lot's of spam looks like it has keywords, but he source actually reviels that the keyword is broken up with html tags and other invisible garbage.

 

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 27 June 2004 at 9:45pm

Alan,

If you 're not able to see the original source of the email, you cannot say "then why does the spam get through when it contains several triggers that my filters would normally have caught", as the email may be formatted in such a way to make your keyword list fail.

Matt has a very good suggestion in this thread. If you are able to finally see the email source we'll be able to see if there's actually a bug in SpamFilter or if the email source is indeed formatted in such a way to byspass your keywords.

Roberto F.
LogSat Software

Back to Top
Alan View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Alan Quote  Post ReplyReply Direct Link To This Post Posted: 28 June 2004 at 3:51pm

I downloaded the SpamSource add-on and all it appears to do is send back a copy of the email with the headers included.   Since the original email came re-encapsulated as an attachement to an email with a body of "This message uses a character set that is not supported by the Internet Service. To view the original message content, open the attached message. If the text doesn't display correctly, save the attachment to disk, and then open it using a viewer that can display the original character set. " thus none of the attachment containing the original email text was included in this apps re-mailing.

Sigh...

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 28 June 2004 at 11:23pm

Alan,

Luck is not with you... One thing you may want to try is the "debug view" in SpamFilter. If you know the IP address of the sender's server, under the "Settings" and then "Debug View" you can try monitoring traffic from that IP. SpamFilter will catch the initial SMTP traffic, and some of the content. Luck will play its part though, since SpamFilter will try to catch as much traffic as possible, but for performance reasons it won't try super hard, and may skip a few packets. What you'll see though is the email's source, or part of it.

Roberto F.
LogSat Software

Back to Top
Alan View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Alan Quote  Post ReplyReply Direct Link To This Post Posted: 30 June 2004 at 12:54pm

Roberto,

can it be that my issue is related to Bill's issue?

http://www.logsat.com/spamfilter/forums/showmessage.asp?messageID=3850

I am wondering if maybe the original contents being converted to an attachment may be what is allowing the emails to get through.

Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 01 July 2004 at 12:15am

Alan,

I've seen that "conversion" on emails received by both Exchange 5.5 and Exchange 2003, both without running SpamFilter. Have you tried the debugging procedure I described in a previous posting to see if you're able to capture the original source?

Roberto F.
LogSat Software

Back to Top
Alan View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Alan Quote  Post ReplyReply Direct Link To This Post Posted: 01 July 2004 at 12:28pm
No unfortunately the emails come in from different IP's so there really isn't one that I can monitor
Back to Top
mikek View Drop Down
Senior Member
Senior Member
Avatar

Joined: 22 February 2005
Location: Switzerland
Status: Offline
Points: 133
Post Options Post Options   Thanks (0) Thanks(0)   Quote mikek Quote  Post ReplyReply Direct Link To This Post Posted: 06 October 2005 at 2:54am
Has filtering by character set been implemented yet?
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 06 October 2005 at 10:11pm
Not yet, but it's very close to the top of the wish list


Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Marco View Drop Down
Senior Member
Senior Member
Avatar

Joined: 07 June 2005
Location: Netherlands
Status: Offline
Points: 137
Post Options Post Options   Thanks (0) Thanks(0)   Quote Marco Quote  Post ReplyReply Direct Link To This Post Posted: 01 November 2005 at 4:22am

Some of the foreign charset mails are beeing blocked by MAPS, and placed in qdb.

i want to check them out for keyword filtering, honeypot etc, but when i dblclick them to investigate i get:

11/01/05 10:12:57:580 -- Exception occurred during DBGridQuarantineDblClick: Read Timeout

All other qdb items work fine, those don't, Roberto, are you aware of this?

I don't want to send the mails through to the adressee, because then the sender gets whitelisted, and i have to dig through the whitelist to remove it.

In cases like this i could use a 'deliver once' button in the qdb gui.

regardless, the error msg i get isnt supposed to happen.

At the moment i have 3 of the foreign sets mails in the db, and all behave the same.

 

Anyone who is capable of getting himself made president, should on no account be allowed to do the job. D.Adams
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 01 November 2005 at 5:18pm
Marco,

Actually that's news to us. If you can send us the full text contents of one of those messages from the tblMsgs table in the database, we'll try to reproduce it. If you have problems extracting the data, please let us know what database platform you're using so we can help you with the process.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
Marco View Drop Down
Senior Member
Senior Member
Avatar

Joined: 07 June 2005
Location: Netherlands
Status: Offline
Points: 137
Post Options Post Options   Thanks (0) Thanks(0)   Quote Marco Quote  Post ReplyReply Direct Link To This Post Posted: 02 November 2005 at 8:00am

mail sent, hope you find something suspicious.

Kind regards,

 

Marco

ps: running SPF build 487, on winNT 4 SP6a server, qdb is running on msAcess DB, using the jet engine



Edited by Marco
Anyone who is capable of getting himself made president, should on no account be allowed to do the job. D.Adams
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 02 November 2005 at 11:09am

Just another thing to think about.

We had an instance where certain spam was getting through to our users and we did not understand why SF was not stopping it. We then realized the spammers were ignoring the MX records for our domains and sending directly to the mail server itself. SF was setup with the IP of the MX records. We kept the mailservers port 25 open for our customers to user to authenticate and send outgoing emails, but the spammers were blowing right by SF by ignoring the 'rules' and were NOT using MX records to send but going straight to the A record. We now have rules on the mailserver to prevent this, but it was a mystery for a while and something to keep in mind when you get some persistent spam traffic that makes no sense.

http://www.webguyz.net
Back to Top
Web123 View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Web123 Quote  Post ReplyReply Direct Link To This Post Posted: 02 November 2005 at 11:23am

We are facing the same problem!

Does anybody know how to accept mail only from certain IPs(SFs) with sendmail? Maby using procmail?

/Web123

Back to Top
Marco View Drop Down
Senior Member
Senior Member
Avatar

Joined: 07 June 2005
Location: Netherlands
Status: Offline
Points: 137
Post Options Post Options   Thanks (0) Thanks(0)   Quote Marco Quote  Post ReplyReply Direct Link To This Post Posted: 03 November 2005 at 4:53am

If you have a separate server for your outgoing mails i would suggest installing a firewall on, or in front of, the receiving mailserver that blocks all but internal network ip's.

Anyone who is capable of getting himself made president, should on no account be allowed to do the job. D.Adams
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.328 seconds.