Queuing Quarantined msgs w/ database down |
Post Reply ![]() |
Author | |
Derk ![]() Groupie ![]() Joined: 26 January 2005 Location: United States Status: Offline Points: 39 |
![]() ![]() ![]() ![]() ![]() Posted: 02 September 2007 at 4:58pm |
We understand that while the database is off-line that messages the spam filter considers to be "bad" are just plain lost. In a previous post that is considered to be a "feature". We don't see how this could possibly be a positive feature. While the spam filter does a great job, it is not perfect and there are often false positives. So if the database is down those messages are just gone-too-bad-hope-you-didn't-want-them!
On the flip side if the destination mail server is down the messages are queued and delivered when the server returns. If these messages can be queued why can't the quarantine messages to be queued or at least give us the option? If not, what about a secondary database if the primary fails? if not, what about the option to forward all messages to the destination if the database is down? Personally, I would rather get garbage delivered and than not get legitimate messages. we're curious how other users feel about the "feature". |
|
![]() |
|
Vermin ![]() Newbie ![]() Joined: 25 January 2006 Location: South Africa Status: Offline Points: 2 |
![]() ![]() ![]() ![]() ![]() |
I would also prefer a option or someway of storing the messages - until the database is back up - we get blamed for loosing anything so always good to have proof of some sort. |
|
![]() |
|
Web123 ![]() Newbie ![]() ![]() Joined: 26 January 2005 Location: Finland Status: Offline Points: 31 |
![]() ![]() ![]() ![]() ![]() |
Me too!
Have tried to convince Roberto into this....maybe now?
/Kim
|
|
![]() |
|
LogSat ![]() Admin Group ![]() ![]() Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
![]() ![]() ![]() ![]() ![]() |
We're watching the thread...
Weve always been hesitant in queing emails if the DB was offline, to then resume the archival when it came back online. This is because there could be a HUGE amount of queued emails that has to e uploaded to the database, and we are afraid that this traffic cold overwhelm the DB server. Throttling the amount of upload to the DB is not simple, as what will fit one customer will not work for others. This said, is a compromise reachable? What if SpamFilter simply spooled the emails to be archived to a temporary folder if the database is offline? That wold allow admis the ability to manually retrieve an email if needed. It wold also be up to the admins to maintain the temp directory by deleting old emails from it... |
|
![]() |
|
Web123 ![]() Newbie ![]() ![]() Joined: 26 January 2005 Location: Finland Status: Offline Points: 31 |
![]() ![]() ![]() ![]() ![]() |
...an option to queue them to a directory is much better than just deleting them |
|
![]() |
|
WebGuyz ![]() Senior Member ![]() Joined: 09 May 2005 Location: United States Status: Offline Points: 348 |
![]() ![]() ![]() ![]() ![]() |
How about a settable throttle where you could set it at a low number for uploading of catch up quaratine emails during peak periods and set it higher during slower periods to try and catchup faster. Just losing quarantined mail if the DB is down could cause a lot of screaming by our customers.
|
|
http://www.webguyz.net
|
|
![]() |
|
Desperado ![]() Senior Member ![]() ![]() Joined: 27 January 2005 Location: United States Status: Offline Points: 1143 |
![]() ![]() ![]() ![]() ![]() |
My 2 cents. Spool them to disk and let the Administrator write a stored procedure to do what ever they want at what ever rate. Offload the entire quarantine process for spooled messages (if SQL is down) away from the spamfilter application and put it in the hands of the SQL server and the Administrator.
|
|
The Desperado
Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
|
![]() |
|
Derk ![]() Groupie ![]() Joined: 26 January 2005 Location: United States Status: Offline Points: 39 |
![]() ![]() ![]() ![]() ![]() |
Any option besides just dumping the message is better than nothing!
|
|
![]() |
|
atifghaffar ![]() Senior Member ![]() ![]() Joined: 31 May 2006 Location: Switzerland Status: Offline Points: 104 |
![]() ![]() ![]() ![]() ![]() |
Roberto, yes. We can write script to read this directory and upload to the datbase when needed without overloading the db. PS: I am asuming that SF-HEADERS will be present to they can be extracted and inserted in the db accordingly. I can provide a perl re-inject script to do this if you provide the spooling functionality. |
|
best regards
Atif |
|
![]() |
|
caratking ![]() Groupie ![]() Joined: 13 March 2006 Location: United States Status: Offline Points: 79 |
![]() ![]() ![]() ![]() ![]() |
I tried to get the concept of not just deleting messages into the product while SFE was being developed... I think it made it onto some wish list, but never got into the final product.
The logic I find most interesting is how SFE was designed to work from the ground up with the thought that the DB would be down at times. By keeping the config / settings locally and always getting updates from the DB. Totally written to handle the failure of a DB. Then to turn around and just delete messages when the DB is down... does not seem right. I vote for placing them in a que, with a configuration setting to throttle how fast they are placed in the DB when the DB comes back online. X number per minute etc. If the DB server is just rebooted, or hangs for a short time there should not be so much mail to process and probably no one would even notice. If someone ends up with 50,000 messages in the queue because the server was off over a weekend they can adjust the insert speed, review & delete confirmed spam etc. to help clear the que. |
|
![]() |
|
Derk ![]() Groupie ![]() Joined: 26 January 2005 Location: United States Status: Offline Points: 39 |
![]() ![]() ![]() ![]() ![]() |
has there been any more thought about this?
|
|
![]() |
|
LogSat ![]() Admin Group ![]() ![]() Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
![]() ![]() ![]() ![]() ![]() |
We are testing a beta version with this feature internally with pretty good results. Would you be interested in testing it as well?
|
|
![]() |
|
Derk ![]() Groupie ![]() Joined: 26 January 2005 Location: United States Status: Offline Points: 39 |
![]() ![]() ![]() ![]() ![]() |
absolutely.
|
|
![]() |
|
LogSat ![]() Admin Group ![]() ![]() Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
![]() ![]() ![]() ![]() ![]() |
I'll be sending you a PM shortly with the download URL.
As far as configurations, there's none needed. SpamFilter will spool emails to be archived in the \SpamFilter\quarantine directory. On successful archival, the files will be deleted. If the database operation fails, the files will be renamed and SpamFilter will re-attempt delivery every 5 minutes. Each email will consist of two files in plain text so they can be examined if needed. One will be the email body itself, the other will contain a few database-specific entries. If the DB is offline for several hours, there will be quite a bit of emails to be inserted into the database. SpamFilter will "throttle" the archival process so that the number of concurrent connections into the database is one half the maximum number of connections allowed nto SpamFilter. |
|
![]() |
|
mikek ![]() Senior Member ![]() ![]() Joined: 22 February 2005 Location: Switzerland Status: Offline Points: 133 |
![]() ![]() ![]() ![]() ![]() |
me too, me too! :-)
while our database is pretty stable, we do often have the problem that the tables are locked by the "delete old quarantine items" process, so queuing would help us a lot in that case! ![]() |
|
![]() |
|
LogSat ![]() Admin Group ![]() ![]() Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
![]() ![]() ![]() ![]() ![]() |
Just sent you a PM as well with the download URL...
|
|
![]() |
|
mikek ![]() Senior Member ![]() ![]() Joined: 22 February 2005 Location: Switzerland Status: Offline Points: 133 |
![]() ![]() ![]() ![]() ![]() |
Thanks, Roberto!
Queueing works, but the .info files stay in the quarantine queue and I can't seem to find any of the queued items in the quarantine db... will check the log and keep you updated. ![]() ![]() Edited by mikek - 08 October 2007 at 8:43am |
|
![]() |
|
mikek ![]() Senior Member ![]() ![]() Joined: 22 February 2005 Location: Switzerland Status: Offline Points: 133 |
![]() ![]() ![]() ![]() ![]() |
10.08.07 14:13:12:252 -- (3976) Starting to process quarantine directory...
was the only line in the log I could find... ![]() |
|
![]() |
|
LogSat ![]() Admin Group ![]() ![]() Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
![]() ![]() ![]() ![]() ![]() |
I apologize for the early beta.. Some last-minute changes in the nightly builds had some less-than-desired side effects. We saw the let-over .info files as well, along with the fact that some queued emails are still not being quarantined.
We've just uploaded an updated beta in the same location I sent in the PM. Just update the build number from 724 to 725 in the URL and you will be ale to download it. You can delete any .info files that are left over from yesterday's build. |
|
![]() |
|
mikek ![]() Senior Member ![]() ![]() Joined: 22 February 2005 Location: Switzerland Status: Offline Points: 133 |
![]() ![]() ![]() ![]() ![]() |
Hi Roberto
Build 725 is running smoothly! As far as I can tell, the queuing and delayed inserting is working as intended. Thanks for a great feature! Now, about that greylisting feature... :-) ![]() |
|
![]() |
|
atifghaffar ![]() Senior Member ![]() ![]() Joined: 31 May 2006 Location: Switzerland Status: Offline Points: 104 |
![]() ![]() ![]() ![]() ![]() |
Roberto,
approximately when will be this feature available? My logs make me look bad... see. mysql> select * from stats_quar_count; +----+------------+--------+ | id | date | count | +----+------------+--------+ | 1 | 2007-09-13 | 516335 | ... ... | 15 | 2007-09-27 | 273103 | | 16 | 2007-09-28 | 180868 | No quarantine for 29,30,01 :-( | 17 | 2007-10-02 | 266506 | | 18 | 2007-10-03 | 451231 | | 19 | 2007-10-04 | 476447 | | 20 | 2007-10-05 | 457875 | | 21 | 2007-10-06 | 423162 | | 22 | 2007-10-07 | 409856 | | 23 | 2007-10-08 | 483850 | | 24 | 2007-10-09 | 511022 | +----+------------+--------+ |
|
best regards
Atif |
|
![]() |
|
LogSat ![]() Admin Group ![]() ![]() Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
![]() ![]() ![]() ![]() ![]() |
Atif,
We've just upgraded the status from beta to pre-release for this latest 3.5.4.725 build, and have made it available in the registered user area. There are currently no known issues with it (and hope it will remain like this ![]() |
|
![]() |
|
atifghaffar ![]() Senior Member ![]() ![]() Joined: 31 May 2006 Location: Switzerland Status: Offline Points: 104 |
![]() ![]() ![]() ![]() ![]() |
Roberto,
Thanks. I will ask our sysadmin to test it soon. Currently I am a bit busy getting around the limbo blacklist cache not available to mortals... :-) |
|
best regards
Atif |
|
![]() |
Post Reply ![]() |
|
Tweet
|
Forum Jump | Forum Permissions ![]() You cannot post new topics in this forum You cannot reply to topics in this forum You cannot delete your posts in this forum You cannot edit your posts in this forum You cannot create polls in this forum You cannot vote in polls in this forum |
This page was generated in 0.180 seconds.