Queuing Quarantined msgs w/ database down
Printed From: LogSat Software
Category: Spam Filter ISP
Forum Name: Spam Filter ISP Support
Forum Description: General support for Spam Filter ISP
URL: https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6225
Printed Date: 05 February 2025 at 10:58am
Topic: Queuing Quarantined msgs w/ database down
Posted By: Derk
Subject: Queuing Quarantined msgs w/ database down
Date Posted: 02 September 2007 at 4:58pm
|
We understand that while the database is off-line that messages the spam filter considers to be "bad" are just plain lost. In a previous post that is considered to be a "feature". We don't see how this could possibly be a positive feature. While the spam filter does a great job, it is not perfect and there are often false positives. So if the database is down those messages are just gone-too-bad-hope-you-didn't-want-them! On the flip side if the destination mail server is down the messages are queued and delivered when the server returns. If these messages can be queued why can't the quarantine messages to be queued or at least give us the option? If not, what about a secondary database if the primary fails? if not, what about the option to forward all messages to the destination if the database is down? Personally, I would rather get garbage delivered and than not get legitimate messages. we're curious how other users feel about the "feature". |
Replies:
Posted By: Vermin
Date Posted: 03 September 2007 at 9:20am
|
I would also prefer a option or someway of storing the messages - until the database is back up - we get blamed for loosing anything so always good to have proof of some sort. |
Posted By: Web123
Date Posted: 03 September 2007 at 10:36am
|
Me too!
Have tried to convince Roberto into this....maybe now?
/Kim
|
Posted By: LogSat
Date Posted: 04 September 2007 at 11:30pm
|
We're watching the thread... Weve always been hesitant in queing emails if the DB was offline, to then resume the archival when it came back online. This is because there could be a HUGE amount of queued emails that has to e uploaded to the database, and we are afraid that this traffic cold overwhelm the DB server. Throttling the amount of upload to the DB is not simple, as what will fit one customer will not work for others. This said, is a compromise reachable? What if SpamFilter simply spooled the emails to be archived to a temporary folder if the database is offline? That wold allow admis the ability to manually retrieve an email if needed. It wold also be up to the admins to maintain the temp directory by deleting old emails from it... ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: Web123
Date Posted: 05 September 2007 at 12:18am
|
...an option to queue them to a directory is much better than just deleting them |
Posted By: WebGuyz
Date Posted: 05 September 2007 at 1:35am
How about a settable throttle where you could set it at a low number for uploading of catch up quaratine emails during peak periods and set it higher during slower periods to try and catchup faster. Just losing quarantined mail if the DB is down could cause a lot of screaming by our customers. ------------- http://www.webguyz.net |
LogSat wrote:Posted By: Desperado
Date Posted: 05 September 2007 at 10:56am
|
My 2 cents. Spool them to disk and let the Administrator write a stored procedure to do what ever they want at what ever rate. Offload the entire quarantine process for spooled messages (if SQL is down) away from the spamfilter application and put it in the hands of the SQL server and the Administrator. ------------- The Desperado Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
Posted By: Derk
Date Posted: 05 September 2007 at 3:16pm
| Any option besides just dumping the message is better than nothing! |
Posted By: atifghaffar
Date Posted: 05 September 2007 at 4:15pm
Roberto, yes. We can write script to read this directory and upload to the datbase when needed without overloading the db. PS: I am asuming that SF-HEADERS will be present to they can be extracted and inserted in the db accordingly. I can provide a perl re-inject script to do this if you provide the spooling functionality. ------------- best regards Atif |
Posted By: caratking
Date Posted: 07 September 2007 at 10:16am
|
I tried to get the concept of not just deleting messages into the product while SFE was being developed... I think it made it onto some wish list, but never got into the final product. The logic I find most interesting is how SFE was designed to work from the ground up with the thought that the DB would be down at times. By keeping the config / settings locally and always getting updates from the DB. Totally written to handle the failure of a DB. Then to turn around and just delete messages when the DB is down... does not seem right. I vote for placing them in a que, with a configuration setting to throttle how fast they are placed in the DB when the DB comes back online. X number per minute etc. If the DB server is just rebooted, or hangs for a short time there should not be so much mail to process and probably no one would even notice. If someone ends up with 50,000 messages in the queue because the server was off over a weekend they can adjust the insert speed, review & delete confirmed spam etc. to help clear the que. |
Posted By: Derk
Date Posted: 07 October 2007 at 4:38pm
| has there been any more thought about this? |
Posted By: LogSat
Date Posted: 07 October 2007 at 4:45pm
|
We are testing a beta version with this feature internally with pretty good results. Would you be interested in testing it as well?
------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: Derk
Date Posted: 07 October 2007 at 5:12pm
| absolutely. |
Posted By: LogSat
Date Posted: 07 October 2007 at 11:57pm
|
I'll be sending you a PM shortly with the download URL. As far as configurations, there's none needed. SpamFilter will spool emails to be archived in the \SpamFilter\quarantine directory. On successful archival, the files will be deleted. If the database operation fails, the files will be renamed and SpamFilter will re-attempt delivery every 5 minutes. Each email will consist of two files in plain text so they can be examined if needed. One will be the email body itself, the other will contain a few database-specific entries. If the DB is offline for several hours, there will be quite a bit of emails to be inserted into the database. SpamFilter will "throttle" the archival process so that the number of concurrent connections into the database is one half the maximum number of connections allowed nto SpamFilter. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: mikek
Date Posted: 08 October 2007 at 2:13am
|
me too, me too! :-) while our database is pretty stable, we do often have the problem that the tables are locked by the "delete old quarantine items" process, so queuing would help us a lot in that case! 
|
Posted By: LogSat
Date Posted: 08 October 2007 at 6:57am
|
Just sent you a PM as well with the download URL... ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: mikek
Date Posted: 08 October 2007 at 8:24am
|
Thanks, Roberto! Queueing works, but the .info files stay in the quarantine queue and I can't seem to find any of the queued items in the quarantine db... will check the log and keep you updated.
|
Posted By: mikek
Date Posted: 08 October 2007 at 8:38am
|
10.08.07 14:13:12:252 -- (3976) Starting to process quarantine directory... was the only line in the log I could find...
|
Posted By: LogSat
Date Posted: 08 October 2007 at 7:59pm
|
I apologize for the early beta.. Some last-minute changes in the nightly builds had some less-than-desired side effects. We saw the let-over .info files as well, along with the fact that some queued emails are still not being quarantined. We've just uploaded an updated beta in the same location I sent in the PM. Just update the build number from 724 to 725 in the URL and you will be ale to download it. You can delete any .info files that are left over from yesterday's build. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: mikek
Date Posted: 09 October 2007 at 3:48am
|
Hi Roberto Build 725 is running smoothly! As far as I can tell, the queuing and delayed inserting is working as intended. Thanks for a great feature! Now, about that greylisting feature... :-)
|
Posted By: atifghaffar
Date Posted: 09 October 2007 at 6:01pm
|
Roberto, approximately when will be this feature available? My logs make me look bad... see. mysql> select * from stats_quar_count; +----+------------+--------+ | id | date | count | +----+------------+--------+ | 1 | 2007-09-13 | 516335 | ... ... | 15 | 2007-09-27 | 273103 | | 16 | 2007-09-28 | 180868 | No quarantine for 29,30,01 :-( | 17 | 2007-10-02 | 266506 | | 18 | 2007-10-03 | 451231 | | 19 | 2007-10-04 | 476447 | | 20 | 2007-10-05 | 457875 | | 21 | 2007-10-06 | 423162 | | 22 | 2007-10-07 | 409856 | | 23 | 2007-10-08 | 483850 | | 24 | 2007-10-09 | 511022 | +----+------------+--------+ ------------- best regards Atif |
Posted By: LogSat
Date Posted: 09 October 2007 at 6:51pm
|
Atif, We've just upgraded the status from beta to pre-release for this latest 3.5.4.725 build, and have made it available in the registered user area. There are currently no known issues with it (and hope it will remain like this  ).------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: atifghaffar
Date Posted: 16 October 2007 at 5:06pm
|
Roberto, Thanks. I will ask our sysadmin to test it soon. Currently I am a bit busy getting around the limbo blacklist cache not available to mortals... :-) ------------- best regards Atif |