Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - Latest Blocking Stats
  FAQ FAQ  Forum Search   Register Register  Login Login

Latest Blocking Stats

 Post Reply Post Reply
Author
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Topic: Latest Blocking Stats
    Posted: 26 January 2006 at 11:34am

All,

For anyone who may be interested, here is a spread of the "actions" and the "reasons" on one of my servers.  This was parsed using "SawMill" and my latest "Plugin" filter for that product. Note that some "reasons" are actually allows rather than blocks --- Marked in green font:

  Action Messages Bytes
1 quarantined 2,621,432 47.8 % 10.38 G
2 Dropped Connection 1,111,740 20.3 % 3.01 G
3 Bypassed all rules 656,438 12.0 % 12.10 G
4 accepted 455,832 8.3 % 21.77 G
5 rejected 375,664 6.8 % 1.28 G
6 sent to NULL 222,836 4.1 % 40.06 M
7 spam-tagged 42,984 0.8 % 263.21 M
  Total 5,486,926 100 %   48.84 G

  Reason Messages Bytes
1 SPF test 839,545 15.3 % 1.51 G
2 Reverse DNS not found 737,478 13.4 % 2.82 G
3 Blacklisted by sbl-xbl.spamhaus.org. 712,997 13.0 % 2.94 G
4 IP is in local blacklist cache 563,403 10.3 % 25.00 k
5 Whitelisted EMail Address To 552,158 10.1 % 6.79 G
6 Probe or Unknown 447,740 8.2 % 3.05 G
7 was queued 446,509 8.1 % 21.73 G
8 Blacklisted by dnsbl.sorbs.net. 306,036 5.6 % 1.93 G
9 EmailTO is in local blacklist file 199,876 3.6 % 32.94 M
10 Blacklisted by multi.surbl.org. 136,413 2.5 % 562.01 M
11 Invalid MX record 100,755 1.8 % 597.42 M
12 Too many connections 85,403 1.6 % 32.17 M
13 Whitelisted EMail Address From 65,414 1.2 % 1.12 G
14 EmailFrom is in local blacklist file 56,690 1.0 % 205.57 M
15 Blacklisted by bl.spamcop.net. 50,614 0.9 % 352.07 M
16 content filter 37,304 0.7 % 406.44 M
17 AutoWhiteList Force Delivery 31,941 0.6 % 4.19 G
18 Exceeded maximum number of RCPT TO 29,487 0.5 % 160.29 M
19 no relay allowed 24,024 0.4 % 3.20 M
20 infected with the virus 20,963 0.4 % 69.55 M
21 IP address is from a blacklisted country 11,789 0.2 % 28.17 M
22 IP is in local blacklist file 5,828 0.1 % 42.28 M
23 IP in local Blacklist 5,038 0.1 % 19.63 M
24 Blacklisted by dnsbl.njabl.org. 4,387 0.1 % 36.25 M
25 Blacklisted by dnsbl.mags.net. 4,075 0.1 % 112.51 M
26 Blocked by Honeypot Autofilter 3,972 0.1 % 34.09 M
27 Blacklisted by combined.njabl.org. 2,535 0.1 % 11.03 M
28 No Data Received 2,030 0.0 % 3.11 M
29 Whitelisted Peer IP 1,001 0.0 % 3.67 M
30 Domain is in local blacklist file 434 0.0 % 5.28 M
31 Blacklisted by dynablock.njabl.org. 237 0.0 % 679.00 k
  Total 5,486,076 100 %   48.74 G



Edited by Desperado
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 28 January 2006 at 11:42am

Desperado,

 I remember your mentioning you use multiple copies of SF. How do you synch up autowhitelistdelivery.txt files (assuming you have 2 boxes doing MX for your domains).

SF works great for us now but if we were to double or triple our size the current anti-spam solution would need to be re-evaluated.

Things I think could become a concern are;

A single autohwitelistdelivery.txt file. (maybe break it up by domain and have SF check each domain list for change and reload if needed. memory and cpu would not be an issue as this would probably be a dedicated box since its in a larger environment)

Sharing of blacklists and whitelists (they synch up or put them on a single shared data drive, but then again, if that shared drive fails your entire antispam system fails. I think every server running SF should have its own set of files but synced periodically over time.

Maybe have a version of SF (call it Multi-SF ;-) that costs a little more per server but gives us an upgrade path in case we get lucky enough to grow.

The thought of having to go through all the pain of evaluating new anti-spam products scares the heck out of me and I'm really happy with SF today, but can see a point in time where the current single server centric model could become a liability.

Anyone else have any thoughts on their being a need for a 'Multi-SF" version?

 

http://www.webguyz.net
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 28 January 2006 at 12:00pm
WebGuyz,

Please note that we license SpamFilter on a per-server basis, which means that once you purchase a license, you can run as many instances of SpamFilter as you like on the same server. Each would be using a different port and/or IP so would all be independent.

Going back to your question regarding sharing the whitelist/blacklist files, whether the installs are on a single server or separate servers, perhaps the simplest way to proceed is with Window's File Replication. If you configure the folders containing the black/white lists on each SpamFilter installation for file replication, each time *any* of these files is updated, the updated file will be pushed to the other directories. SpamFilter continuosly monitors is configuration files for changes, and when a change does occur, the updated file is automatically re-imported. This allows each SpamFilter instance to have its "local" files, and also allows any change on any SpamFilter file to be propagated to the other installs.

We have in the past tried to store the black/white list files in a database, but the performance we obtained was very poor compared to using text files. We may revisit this option in the future, by storing the original settings in the database and then flushing them periodically to files, and have SpamFilter then use these files.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 28 January 2006 at 12:53pm

Roberto,

   Good idea about the replication. The autowhitelistdelivery.txt then becomes the bottleneck for us. We are almost at 3 meg and wonder when that size gets too big.

Another concern is the authowhitelistdelivery.txt can not be mirrored because when using the quantine db and an entry gets tagged for delivery, which server (assuming 2 for now) would get tasked with updating the quarantine db (assuming you have 1 db) and delivering the email into the queue. Would you mirror this file on size, latest date? If 2 different users released 2 different emails and they were handled by 2 different servers at about the same time, then updating can become an issue.

Any thought to splitting up the autowhitelistdelivery.txt file into multiple files that only get loaded every minute if changed? Maybe along domain names boundries?

Thanks for a great product, just trying to make sure it can grow along with us.

  

http://www.webguyz.net
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 28 January 2006 at 1:58pm
The database should not be an issue. We do support multiple instances of SpamFilter all sharing the same database. This happens in the background, we don't "bother" administrators with the setup, but everytime you install a new SpamFilter, the instance adds itself to the "tblServers" table in the database. This table contains all the SpamFilter instances that connect to it. Furthermore, each quarantined message has a "ServerID" field to mark which SpamFilter instance has quarantined that message. When a message is tagged by a user for delivery, thanks to this "mark", the correct SpamFilter server that originally quarantined the message will be delivering it.

There should not be problems with multiple updates tothe autowhitelistdelivery.txt files, as long as the changes are not concurrent, as Window's File Replication should be able to support changes made to any of its folders.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 28 January 2006 at 2:29pm

Then the last  concern is the size of the autowhitelistdelivery.txt file and having only one.

We use it in a non-conventional way by parsing our outgoing email server log for outgoing emails and adding FROM|TO pairs into the autowhitelistdelivery.txt file thereby whitelisting the addresses of the customers our users send emails to. Its doing good for now but the question is what number will break the camels back as far as size.

http://www.webguyz.net
Back to Top
kwikstix View Drop Down
Newbie
Newbie
Avatar

Joined: 31 January 2006
Location: United States
Status: Offline
Points: 5
Post Options Post Options   Thanks (0) Thanks(0)   Quote kwikstix Quote  Post ReplyReply Direct Link To This Post Posted: 01 February 2006 at 10:01am

Desperado,

I've been using sawmill for my website stats for some time.  I'm currently running the latest production version (7.2), but my SF stats look different than yours.  Primarily, I only get two Actions in my report: Accepted and Rejected.  Do I have something misconfigured, or does Sawmill not include the same config you're using?

Mike

Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 01 February 2006 at 10:15am

KwikStix,

 

Please look at the log format plugin and tell me what version that is.  It is in the LogAnalysisInfo/log_formats/logsat_spam_filter_isp.cfg file and should read:

# Updated to match new log entries - Dec-30-2005 Dan Seligmann, Mags Net, LLC
log.format.format_label = "LogSat SpamFilterISP Log Format B500.8"

The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
kwikstix View Drop Down
Newbie
Newbie
Avatar

Joined: 31 January 2006
Location: United States
Status: Offline
Points: 5
Post Options Post Options   Thanks (0) Thanks(0)   Quote kwikstix Quote  Post ReplyReply Direct Link To This Post Posted: 01 February 2006 at 10:20am

Desperado,

The date on my config file is May 20, 2005

Mike

Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 01 February 2006 at 10:32am
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
kwikstix View Drop Down
Newbie
Newbie
Avatar

Joined: 31 January 2006
Location: United States
Status: Offline
Points: 5
Post Options Post Options   Thanks (0) Thanks(0)   Quote kwikstix Quote  Post ReplyReply Direct Link To This Post Posted: 01 February 2006 at 10:43am

Wow - quick responses - you ROCK man!

Thanks for the updated cfg file.  However, I'm still only seeing Accepted and Rejected under my Actions report in Sawmill.  Is something not being logged correctly in SF?

Mike

Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 01 February 2006 at 10:53am

Kwikstix,

Here is the problem ... even if you overwrite the log plug-in, you have to delete and re-create the "Profile" or the Sawmill profile will continue to use the existing (old) format.  This means you have to re-import all your logs.  This was a pain in the arse for me when I update my log format but I deal with it.

Please try that and let me know if you still get "wimpy" data.  You should get up to 7 actions and hmmmm, around 20-35 reasons depending on your SpamFilter options and setup.

Please get back to me as I want to make sure this is working for ALL installs.  BTW, which SpamFilter version are you running?  Older versions may not parse as well.

The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
kwikstix View Drop Down
Newbie
Newbie
Avatar

Joined: 31 January 2006
Location: United States
Status: Offline
Points: 5
Post Options Post Options   Thanks (0) Thanks(0)   Quote kwikstix Quote  Post ReplyReply Direct Link To This Post Posted: 01 February 2006 at 11:29am

Desperado - It WORKED!  Thanks a million!

You asked what version I'm using.  I started evaluating SF last week, so I'm still only running 2.7.1.511 in eval mode.  It hasn't taken me long to know that this is definitely the spam filter I'm gonna buy, though, especially now that I'm sorting out the statistics piece of it.

You're a great resource - thanks a lot!

Mike

Back to Top
Desperado View Drop Down
Senior Member
Senior Member
Avatar

Joined: 27 January 2005
Location: United States
Status: Offline
Points: 1143
Post Options Post Options   Thanks (0) Thanks(0)   Quote Desperado Quote  Post ReplyReply Direct Link To This Post Posted: 01 February 2006 at 11:49am

Kwikstix,

Very glad to hear it works now.  Did you purchase SawMill or are you going to purchase it?  If so, would you mind terribly letting Greg over at SawMill know you are using it for the LogSat logs and saw my postings?  His address is  ferrar at flowerfire dot com. I do not know if it will get me anything but I do bug him a lot and he is VERY helpfull over there and letting him know that his time spent over the last year has actually led to a sale would be nice.

I hope SpamFilterISP works as well for you as it has for me.  I do not think I have EVER gotten the kind of support that Roberto gives us from any other product.  THANKS Roberto ... and don't forget your parachute!

The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Back to Top
kwikstix View Drop Down
Newbie
Newbie
Avatar

Joined: 31 January 2006
Location: United States
Status: Offline
Points: 5
Post Options Post Options   Thanks (0) Thanks(0)   Quote kwikstix Quote  Post ReplyReply Direct Link To This Post Posted: 01 February 2006 at 12:00pm
No problem, Dan.  I have already purchased Sawmill, but next time I check in with Greg (which is fairly often for me too), I'll let him know that you're DA MAN.  Thanks again for your help, and I'm really looking forward to diving in to SF more.  It looks like a GREAT product, and I'm excited about the quality and timeliness of responses in these forums!  I'll echo the thanks to Roberto!!
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.316 seconds.