All,

For anyone who may be interested, here is a spread of the "actions" and the "reasons" on one of my servers.  This was parsed using "SawMill" and my latest "Plugin" filter for that product. Note that some "reasons" are actually allows rather than blocks --- Marked in green font:

	Action	Messages			Bytes
1	quarantined	2,621,432	47.8 %		10.38 G
2	Dropped Connection	1,111,740	20.3 %		3.01 G
3	Bypassed all rules	656,438	12.0 %		12.10 G
4	accepted	455,832	8.3 %		21.77 G
5	rejected	375,664	6.8 %		1.28 G
6	sent to NULL	222,836	4.1 %		40.06 M
7	spam-tagged	42,984	0.8 %		263.21 M
	Total	5,486,926	100 %		48.84 G

	javascript ;" target=_blank onclick="set_sort_column0, 'reason'; return false; - Reason	javascript ;" target=_blank onclick="set_sort_column0, 'messages'; return false; - Messages			javascript ;" target=_blank onclick="set_sort_column0, 'bytes'; return false; - Bytes
1	javascript ;" target=_blank onclick="zoom0, 'reason', 'SPF__HexEsc__20test'; return false; - SPF test	839,545	15.3 %		1.51 G
2	javascript ;" target=_blank onclick="zoom0, 'reason', 'Reverse__HexEsc__20DNS__HexEsc__20not__HexEsc__20found'; return false; - Reverse DNS not found	737,478	13.4 %		2.82 G
3	javascript ;" target=_blank onclick="zoom0, 'reason', 'Blacklisted__HexEsc__20by__HexEsc__20sbl-xbl.spamhaus.org.__HexEsc__20'; return false; - Blacklisted by sbl-xbl.spamhaus.org.	712,997	13.0 %		2.94 G
4	javascript ;" target=_blank onclick="zoom0, 'reason', 'IP__HexEsc__20is__HexEsc__20in__HexEsc__20local__HexEsc__20blacklist__HexEsc__20cache'; return false; - IP is in local blacklist cache	563,403	10.3 %		25.00 k
5	javascript ;" target=_blank onclick="zoom0, 'reason', '__HexEsc__20Whitelisted__HexEsc__20EMail__HexEsc__20Address__HexEsc__20To'; return false; - Whitelisted EMail Address To	552,158	10.1 %		6.79 G
6	javascript ;" target=_blank onclick="zoom0, 'reason', 'Probe__HexEsc__20or__HexEsc__20Unknown'; return false; - Probe or Unknown	447,740	8.2 %		3.05 G
7	javascript ;" target=_blank onclick="zoom0, 'reason', 'was__HexEsc__20queued'; return false; - was queued	446,509	8.1 %		21.73 G
8	javascript ;" target=_blank onclick="zoom0, 'reason', 'Blacklisted__HexEsc__20by__HexEsc__20dnsbl.sorbs.net.__HexEsc__20'; return false; - Blacklisted by dnsbl.sorbs.net.	306,036	5.6 %		1.93 G
9	javascript ;" target=_blank onclick="zoom0, 'reason', 'EmailTO__HexEsc__20is__HexEsc__20in__HexEsc__20local__HexEsc__20blacklist__HexEsc__20file'; return false; - EmailTO is in local blacklist file	199,876	3.6 %		32.94 M
10	javascript ;" target=_blank onclick="zoom0, 'reason', 'Blacklisted__HexEsc__20by__HexEsc__20multi.surbl.org.__HexEsc__20'; return false; - Blacklisted by multi.surbl.org.	136,413	2.5 %		562.01 M
11	javascript ;" target=_blank onclick="zoom0, 'reason', 'Invalid__HexEsc__20MX__HexEsc__20record'; return false; - Invalid MX record	100,755	1.8 %		597.42 M
12	javascript ;" target=_blank onclick="zoom0, 'reason', 'Too__HexEsc__20many__HexEsc__20connections'; return false; - Too many connections	85,403	1.6 %		32.17 M
13	javascript ;" target=_blank onclick="zoom0, 'reason', '__HexEsc__20Whitelisted__HexEsc__20EMail__HexEsc__20Address__HexEsc__20From'; return false; - Whitelisted EMail Address From	65,414	1.2 %		1.12 G
14	javascript ;" target=_blank onclick="zoom0, 'reason', 'EmailFrom__HexEsc__20is__HexEsc__20in__HexEsc__20local__HexEsc__20blacklist__HexEsc__20file'; return false; - EmailFrom is in local blacklist file	56,690	1.0 %		205.57 M
15	javascript ;" target=_blank onclick="zoom0, 'reason', 'Blacklisted__HexEsc__20by__HexEsc__20bl.spamcop.net.__HexEsc__20'; return false; - Blacklisted by bl.spamcop.net.	50,614	0.9 %		352.07 M
16	javascript ;" target=_blank onclick="zoom0, 'reason', 'content__HexEsc__20filter'; return false; - content filter	37,304	0.7 %		406.44 M
17	javascript ;" target=_blank onclick="zoom0, 'reason', '__HexEsc__20AutoWhiteList__HexEsc__20Force__HexEsc__20Delivery'; return false; - AutoWhiteList Force Delivery	31,941	0.6 %		4.19 G
18	javascript ;" target=_blank onclick="zoom0, 'reason', 'Exceeded__HexEsc__20maximum__HexEsc__20number__HexEsc__20of__HexEsc__20RCPT__HexEsc__20TO'; return false; - Exceeded maximum number of RCPT TO	29,487	0.5 %		160.29 M
19	javascript ;" target=_blank onclick="zoom0, 'reason', 'no__HexEsc__20relay__HexEsc__20allowed'; return false; - no relay allowed	24,024	0.4 %		3.20 M
20	javascript ;" target=_blank onclick="zoom0, 'reason', 'infected__HexEsc__20with__HexEsc__20the__HexEsc__20virus'; return false; - infected with the virus	20,963	0.4 %		69.55 M
21	javascript ;" target=_blank onclick="zoom0, 'reason', 'IP__HexEsc__20address__HexEsc__20is__HexEsc__20from__HexEsc__20a__HexEsc__20blacklisted__HexEsc__20country'; return false; - IP address is from a blacklisted country	11,789	0.2 %		28.17 M
22	javascript ;" target=_blank onclick="zoom0, 'reason', 'IP__HexEsc__20is__HexEsc__20in__HexEsc__20local__HexEsc__20blacklist__HexEsc__20file'; return false; - IP is in local blacklist file	5,828	0.1 %		42.28 M
23	javascript ;" target=_blank onclick="zoom0, 'reason', 'IP__HexEsc__20in__HexEsc__20local__HexEsc__20Blacklist'; return false; - IP in local Blacklist	5,038	0.1 %		19.63 M
24	javascript ;" target=_blank onclick="zoom0, 'reason', 'Blacklisted__HexEsc__20by__HexEsc__20dnsbl.njabl.org.__HexEsc__20'; return false; - Blacklisted by dnsbl.njabl.org.	4,387	0.1 %		36.25 M
25	javascript ;" target=_blank onclick="zoom0, 'reason', 'Blacklisted__HexEsc__20by__HexEsc__20dnsbl.mags.net.__HexEsc__20'; return false; - Blacklisted by dnsbl.mags.net.	4,075	0.1 %		112.51 M
26	javascript ;" target=_blank onclick="zoom0, 'reason', 'Blocked__HexEsc__20by__HexEsc__20Honeypot__HexEsc__20Autofilter'; return false; - Blocked by Honeypot Autofilter	3,972	0.1 %		34.09 M
27	javascript ;" target=_blank onclick="zoom0, 'reason', 'Blacklisted__HexEsc__20by__HexEsc__20combined.njabl.org.__HexEsc__20'; return false; - Blacklisted by combined.njabl.org.	2,535	0.1 %		11.03 M
28	javascript ;" target=_blank onclick="zoom0, 'reason', 'No__HexEsc__20Data__HexEsc__20Received'; return false; - No Data Received	2,030	0.0 %		3.11 M
29	javascript ;" target=_blank onclick="zoom0, 'reason', '__HexEsc__20Whitelisted__HexEsc__20Peer__HexEsc__20IP'; return false; - Whitelisted Peer IP	1,001	0.0 %		3.67 M
30	javascript ;" target=_blank onclick="zoom0, 'reason', 'Domain__HexEsc__20is__HexEsc__20in__HexEsc__20local__HexEsc__20blacklist__HexEsc__20file'; return false; - Domain is in local blacklist file	434	0.0 %		5.28 M
31	javascript ;" target=_blank onclick="zoom0, 'reason', 'Blacklisted__HexEsc__20by__HexEsc__20dynablock.njabl.org.__HexEsc__20'; return false; - Blacklisted by dynablock.njabl.org.	237	0.0 %		679.00 k
	Total	5,486,076	100 %		48.74 G

Desperado,

 I remember your mentioning you use multiple copies of SF. How do you synch up autowhitelistdelivery.txt files (assuming you have 2 boxes doing MX for your domains).

SF works great for us now but if we were to double or triple our size the current anti-spam solution would need to be re-evaluated.

Things I think could become a concern are;

A single autohwitelistdelivery.txt file. (maybe break it up by domain and have SF check each domain list for change and reload if needed. memory and cpu would not be an issue as this would probably be a dedicated box since its in a larger environment)

Sharing of blacklists and whitelists (they synch up or put them on a single shared data drive, but then again, if that shared drive fails your entire antispam system fails. I think every server running SF should have its own set of files but synced periodically over time.

Maybe have a version of SF (call it Multi-SF ;-) that costs a little more per server but gives us an upgrade path in case we get lucky enough to grow.

The thought of having to go through all the pain of evaluating new anti-spam products scares the heck out of me and I'm really happy with SF today, but can see a point in time where the current single server centric model could become a liability.

Anyone else have any thoughts on their being a need for a 'Multi-SF" version?

Roberto,

   Good idea about the replication. The autowhitelistdelivery.txt then becomes the bottleneck for us. We are almost at 3 meg and wonder when that size gets too big.

Another concern is the authowhitelistdelivery.txt can not be mirrored because when using the quantine db and an entry gets tagged for delivery, which server (assuming 2 for now) would get tasked with updating the quarantine db (assuming you have 1 db) and delivering the email into the queue. Would you mirror this file on size, latest date? If 2 different users released 2 different emails and they were handled by 2 different servers at about the same time, then updating can become an issue.

Any thought to splitting up the autowhitelistdelivery.txt file into multiple files that only get loaded every minute if changed? Maybe along domain names boundries?

Thanks for a great product, just trying to make sure it can grow along with us.

Then the last  concern is the size of the autowhitelistdelivery.txt file and having only one.

We use it in a non-conventional way by parsing our outgoing email server log for outgoing emails and adding FROM|TO pairs into the autowhitelistdelivery.txt file thereby whitelisting the addresses of the customers our users send emails to. Its doing good for now but the question is what number will break the camels back as far as size.

Desperado,

I've been using sawmill for my website stats for some time.  I'm currently running the latest production version (7.2), but my SF stats look different than yours.  Primarily, I only get two Actions in my report: Accepted and Rejected.  Do I have something misconfigured, or does Sawmill not include the same config you're using?

Mike

KwikStix,

Please look at the log format plugin and tell me what version that is.  It is in the LogAnalysisInfo/log_formats/logsat_spam_filter_isp.cfg file and should read:

# Updated to match new log entries - Dec-30-2005 Dan Seligmann, Mags Net, LLC
log.format.format_label = "LogSat SpamFilterISP Log Format B500.8"

Desperado,

The date on my config file is May 20, 2005

Mike

Get the latest at:

http://spamman.mags.net/sawmill/logsat_spam_filter_isp.cfg - http://spamman.mags.net/sawmill/logsat_spam_filter_isp.cfg

Wow - quick responses - you ROCK man!

Thanks for the updated cfg file.  However, I'm still only seeing Accepted and Rejected under my Actions report in Sawmill.  Is something not being logged correctly in SF?

Mike

Kwikstix,

Here is the problem ... even if you overwrite the log plug-in, you have to delete and re-create the "Profile" or the Sawmill profile will continue to use the existing (old) format.  This means you have to re-import all your logs.  This was a pain in the arse for me when I update my log format but I deal with it.

Please try that and let me know if you still get "wimpy" data.  You should get up to 7 actions and hmmmm, around 20-35 reasons depending on your SpamFilter options and setup.

Please get back to me as I want to make sure this is working for ALL installs.  BTW, which SpamFilter version are you running?  Older versions may not parse as well.

Desperado - It WORKED!  Thanks a million!

You asked what version I'm using.  I started evaluating SF last week, so I'm still only running 2.7.1.511 in eval mode.  It hasn't taken me long to know that this is definitely the spam filter I'm gonna buy, though, especially now that I'm sorting out the statistics piece of it.

You're a great resource - thanks a lot!

Mike

Kwikstix,

Very glad to hear it works now.  Did you purchase SawMill or are you going to purchase it?  If so, would you mind terribly letting Greg over at SawMill know you are using it for the LogSat logs and saw my postings?  His address is  ferrar at flowerfire dot com. I do not know if it will get me anything but I do bug him a lot and he is VERY helpfull over there and letting him know that his time spent over the last year has actually led to a sale would be nice.

I hope SpamFilterISP works as well for you as it has for me.  I do not think I have EVER gotten the kind of support that Roberto gives us from any other product.  THANKS Roberto ... and don't forget your parachute!