Emails Quarantined for Nonexistent users |
Post Reply ![]() |
Author | |
jerbo128 ![]() Senior Member ![]() ![]() Joined: 06 March 2006 Status: Offline Points: 178 |
![]() ![]() ![]() ![]() ![]() Posted: 27 March 2008 at 12:40pm |
I am getting a fair amount of mail in the quarantine for email addresses that do not appear in my tblwl_authorizedtoemails.
I have a script that expires any message in tbl_quarantine where the email field does not have a match in tblwl_authorizedtoemails.
I cleaned such messages this am, and now 2 hours later, I have 2600 more messages that match this rule. The first time that I ran this rule, it removed 200K messages from my database.
After looking at the logs, It appears that the messages are being quarantined because they failed the local blacklist check. So, the authorized To Email list is never checked.
Roberto - is this the intended behavior? What can we do to stop these from clogging my database up?
Running 773 SFE.
Any idea's?
Jeremy Edited by jerbo128 - 27 March 2008 at 2:39pm |
|
![]() |
|
LogSat ![]() Admin Group ![]() ![]() Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
![]() ![]() ![]() ![]() ![]() |
Jeremy,
The order in which the filters is applied is listed at http://www.logsat.com/SpamFilter/Forums/forum_posts.asp?TID=5171#5826. If a filter blocks an email, all subsequent filters will be ignored to avoid wasting resources. This will indeed cause a number of emails where the recipient is not in the "Autorized TO" list to be quarantined instead of being rejected right away. |
|
![]() |
|
jerbo128 ![]() Senior Member ![]() ![]() Joined: 06 March 2006 Status: Offline Points: 178 |
![]() ![]() ![]() ![]() ![]() |
Roberto,
I understand, but in my opinion, this could be a huge isse. With my retention of 7 days - that is 200,000 messages. To me, that is a waste of resources also.
Maybe we need to weigh it out -to see which has a higher "cost"
Anyone else have thoughts on this? How about some of you bigger guys who get a lot of mail, have you ever looked at the number of orphaned messages?
|
|
![]() |
|
LogSat ![]() Admin Group ![]() ![]() Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
![]() ![]() ![]() ![]() ![]() |
jeremy,
The only three filters that are before the "Authorized TO" list are:
emails blocked by these are usually very, very small in numbers, if any at all. Are you actually receiving 200,000 emails each week blocked by these 3? If you have a huge blacklist of blacklisted sender's domains and emails, that may explain this large number, but if not, you may be targeted by very specific spammers/attackers. In "normal" installations, the 3 lists above are often much smaller than the "Authorized TO" list, which is why SpamFilter examines the smaller ones first to try saving some resources. An option may be to change the order of the filters by moving up the "Authorized TO". We'll be monitoring the thread for feedback on this. |
|
![]() |
|
jerbo128 ![]() Senior Member ![]() ![]() Joined: 06 March 2006 Status: Offline Points: 178 |
![]() ![]() ![]() ![]() ![]() |
Local domain blacklist =23,000 rows
Emails to/from = less than 100 each
Local IP blacklist = 250,000 rows (does not count for subnets)
Most of these emails that I am seeing to unAutorized addresses are quarantined with ID 12 - Local IP Blacklist. I see a few with reason 7 and 9.
Yes, you are correct - I am getting over 200,000 per week to these unauthorized addresses. Looking at the IP's and subjects of the connections - they are scattered everywhere.
You said above that local IP blacklist is AFTER the Authorized User Check, then why am I seeing all of these? For clarification -This post lists a different filter order. http://www.logsat.com/SpamFilter/Forums/forum_posts.asp?TID=5171
Which is right?
Jeremy Edited by jerbo128 - 27 March 2008 at 7:20pm |
|
![]() |
|
LogSat ![]() Admin Group ![]() ![]() Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
![]() ![]() ![]() ![]() ![]() |
Geez, I can't even read my own warning. And think that I had even super bolded the notice "Please always check the latest post in this thread" on that thread.
You're right, I did not see the updated order. What you posted is correct, the 4 filters (not 3) that cause emails to be quarantined before the "Authorized TO" list is hit are:
I did not mention the others that appear first (allowed domains, and the "forgotten" blacklist cache and greylist) because these ones reject emails immediately, without allowing them to be quarantined, so do not apply in ths discussion. c |
|
![]() |
|
jerbo128 ![]() Senior Member ![]() ![]() Joined: 06 March 2006 Status: Offline Points: 178 |
![]() ![]() ![]() ![]() ![]() |
Do you have any suggestions for dealing with such a huge amount of crap mail? My SQL server stays much happier with 200K less emails in it's tummy.
|
|
![]() |
|
LogSat ![]() Admin Group ![]() ![]() Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
![]() ![]() ![]() ![]() ![]() |
That depends on the reliability of your IP blacklist, which is causing the bulk of these blocks. If you are note receiving false positives from it, you could enable the option to "do not quarantine" emails from this filter.
Also, if you have the list of allowed users in a database, it's rather simple to create a query that will weed out the unwanted recipients. If the list is only in a text file, there's a few extra steps involved to bulk import it in a temp table in MySQL (don't know yet if MySQL can perform a query directly against a text file, as MS SQL can do). I'll take a stab at it tonite to see if I can help you here, but please email me to stop me if the data is already in the DB! |
|
![]() |
|
jerbo128 ![]() Senior Member ![]() ![]() Joined: 06 March 2006 Status: Offline Points: 178 |
![]() ![]() ![]() ![]() ![]() |
Don't worry about the script. The users are in the DB. And I have a query to weed out the bad emails.
My point is that when they come in at a rate of 30,000 per day, I just see it as a lot of unneeded SQL traffic. Both inbound and then deleting.
As for the reliability of my blacklist, it's pretty good. But when you have that many entries, it only takes 1 bed entry to get a chewing from the customer. So I don't know if I am ready to flip the flag for "do not quarantine"
Jeremy
|
|
![]() |
|
LogSat ![]() Admin Group ![]() ![]() Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
![]() ![]() ![]() ![]() ![]() |
Oh well.. since I've already started, and as it can be used by others who do have a file-based list, here it is anyways :-)
CREATE TABLE test( email varchar(320) ); LOAD DATA INFILE "/tmp/WL_AuthorizedTOEmails.txt" INTO TABLE test LINES TERMINATED BY '\r\n'; UPDATE tblquarantine LEFT JOIN test ON tblquarantine.emailto = test.email SET expire=1 WHERE ISNULL(test.email) DROP TABLE test; The above will mark the emails for deletion so that SpamFilter can then delete them during regular cleanup. If you want to delete them right away, this one will do the job: CREATE TABLE test( email varchar(320) ); LOAD DATA INFILE "/tmp/WL_AuthorizedTOEmails.txt" INTO TABLE test LINES TERMINATED BY '\r\n'; DELETE tblquarantine FROM tblquarantine LEFT JOIN test ON tblquarantine.emailto = test.email WHERE ISNULL(test.email); DELETE tblmsgs FROM tblmsgs LEFT JOIN tblquarantine ON tblmsgs.msgid = tblquarantine.msgid WHERE ISNULL(tblquarantine.msgid); DROP TABLE test; |
|
![]() |
Post Reply ![]() |
|
Tweet
|
Forum Jump | Forum Permissions ![]() You cannot post new topics in this forum You cannot reply to topics in this forum You cannot delete your posts in this forum You cannot edit your posts in this forum You cannot create polls in this forum You cannot vote in polls in this forum |
This page was generated in 0.287 seconds.