We are running the latest version of spamfilter first on a windows 2000 server, then now later on Windows 2003.

We do use quarantene option and we store it for about 10 days.

Then it will delete mails every 30min.

 

I will guess we get about 75000 - 100.000 mails through our filter everyday.
I will guess about 25% goes to quarantene database. This is all Reverse DNS as we have seen that alot of big institutions etc do not use it and we need a tool to fish them up when the filter correctly put them in quarantene.

 

The problems starts after about 3 weeks.

The program starts to freeze and it stops responding.

When we restart the server and start spamfilter, it freezes and only when we disable the quarantene database, then the program works fine.

 

We are using it with mysql 5.0

 

My question is:

Is there a known problem using quarantene database with the quantity of mails I am talking about?

Is it worth trying SQL instead of MySQL?

 

We do not use the quarantene database viewer within the application as we have before been adviced not to use this.

 

Its the 3rd time we have installed the system and it seems to be working fine for about 3 weeks, then we have no other options that turning off the quarantene database to secure that the system will run.

 

Any advice/comments are welcome

 

 

Kind Regards

Morten

There are no known problems relating SpamFilter with MySQL (not MSSQL) and the quarantine database. If you could please zip and provide us with 2 days' worth of SpamFilter logfiles for the day SpamFilter last crash and the day prior to that (we'd like two days in case there's events that lead up to the crash several hours before), we'll try to go thru them to see if we see any problems.

If the logs are over 8MB in size (likely seeing your traffic), I'll be sending you via a PM the ftp login info for out FTP site.

its now on your ftp server: logfile_morten44.txt

 

2 days of log files i 1GB!!!!!! so its a little hard to get it uploaded :)

Unfortunately the log entries you sent do contain several database errors (most of them due to the MySQL Server not being available), but do not show SpamFilter crashing.

In regards to the logfile, if you compress the logfiles with WinZip (or Windows' built-in zip compression), the 1GB logfile sizes will likely shrink down to much less than 100MB making it much easier to transfer (but we ourselves will still have to debug 1GB worth of data....).

In addition to errors in your logs about MySQL being unavailable, we also saw an unusual error occurring when cleaning up the quarantine. The logs report that this SQL statement is invalid:

Would you mind running that SQL statement on your SpamFilter database and let us know if it reports any errors? 

Thanks for your answer.

Yes I can see that I had some issues with Mysql.

I started it and it stopped again straight away.

There where some errors in log file that size of log file was different than specified in .cnf file...

hm did not have a .cnf file, only .ini file. I created one and now the service starts.
Not sure why it worked fine for 1 month, then this happened.

 

Anyway, I did run the command suggested and it did fail

 

See: http://www.e-advice.dk/files/mysql_statement.jpg - http://www.e-advice.dk/files/mysql_statement.jpg

 

We are currently not running ISP spamfilter on this mail server, due to the problem, but guess that should not cause the statement to fail?

 

 

Regarding the log files.
I will try to get them zipped and uploaded some point tomorrow as I can not get to do it at the moment

 

I wonder why it fails?

Wonder if it ever has worked. Would you like some logfiles from when it worked fine as well, just to check if this statement always have failed?

 

Kind Regards

Morten

We've gone thru your files, and found out that the SQL statement that is giving an error:

DELETE DISTINCTROW tblMsgs.* FROM tblMsgs LEFT JOIN tblQuarantine ON tblMsgs.MsgID = tblQuarantine.MsgID WHERE (tblQuarantine.MsgID IS NULL);

is a "generic" SQL statement compatible with MSAccess being executed as a "last resort", since the correct statement that was to be executed for MySQL resulted in an error just prior to that. Seeing the other issues you were having with MySQL this is normal and is not something to worry about.

Going back to the "freezing", from the logs we see that SpamFilter was shut down cleanly at:

07/24/09 00:09:18:343 -- Shutting down all threads. Please wait up to 15-20 seconds....

but something went wrong with the restart, as sometime between 00:14 and 00:20 the SpamFilter process was killed and SpamFilter restarted. We did not see any freezes outside that timeframe. When you say "freeze and stop responding", can you let us know more in detail what exactly you see occurring?

As a side-note, in addition to the issues with MySQL, we saw several out-of-memory errors on the 23rd, occurring when SpamFilter attempted to reload the Bayesian database. Just to be on the safe side, can you please reset the Bayesian database? To do so, just stop SpamFilter, delete (or rename) the SpamFilter/corpus directory, and then restart SpamFilter? That should reset the corpus database for the bayesian filter in case it was corrupted.

I'm afraid we won't be able to answer with 100% certainty here due to the several problem-factors that are involve. let me explain a few issues that when combined as in your case lead to uncertainty.

1. When SpamFilter starts up (as is the case in your screenshot), the GUI will appear frozen and won't refresh until SpamFilter is fully loaded. This can take a few minutes, especially if some of the black/white lists are large (in the order of 1MB or more).

2. If accessing SpamFilter's server via RDP (Remote Desktop/Terminal Services) on a slow connection (< 10Mb/s), and there are several (30/40+) concurrent connections, and you are viewing the Activity Log window, the screen refreshes caused by the activity log window may occur so fast that RDP is not able to keep up, causing the entire RDP session (not just SpamFilter) to appear frozen - mouse clicks and screen refreshes will be extremely slow on the entire RDP window. When this happens, connections may build up on SpamFilter as the software is unable to flush the screen causing internal queues to affect the email handling queues as well. We are aware of this flaw, as RDP issues should not affect email flow, and are working on fixes for it.

3. In the logs we received, MySQL connectivity was not working, causing practically every single email that was rejected to fail the database archiving procedure. When this happens, the quarantined emails are stored in a queue on disk. SpamFilter will attempt to re-archive this emails when the DB is back online, and during SpamFilter's startup, this archival process can use up more resources if there are tens of thousands (or hundreds) of queued emails to scan upon startup. These emails are stored in the \SpamFilter\Quarantine directory. This directory should be empty at all times, with the exception of the handful of files being processed at any instant. If there's tens of thousand or more of files in there, this indicates a database issue.

4. The bayesian database *appeared* corrupted, or there was a low memory issue on the server (corruption was however more likely).

All of these things could have contributed to causing problems. This said, is the issue still occurring? Was MySQL repaired? There are no known issues with MySQL, but if the database itself becomes corrupted then yes, this will affect SpamFilter. Do you have a large amount of files in the \SpamFilter\Quarantine ?

If the problems are still ongoing, are you able to allow us to remote into the server to have a better look?