Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - Idea to reduce database usage
  FAQ FAQ  Forum Search   Register Register  Login Login

Idea to reduce database usage

 Post Reply Post Reply
Author
kspare View Drop Down
Senior Member
Senior Member


Joined: 26 January 2005
Location: Canada
Status: Offline
Points: 334
Post Options Post Options   Thanks (0) Thanks(0)   Quote kspare Quote  Post ReplyReply Direct Link To This Post Topic: Idea to reduce database usage
    Posted: 28 September 2005 at 6:57pm

This will have it's ups and downs, but it's more the idea i'm looking to pursue right now.

When you connect to an smtp server it is possible to verify if the user exists, why couldn't we write a script or integrate this into spamfilter so that it could run maint once a night to check against the database and against users servers to see if users exist? Spamfilter already knows where to send email, either direct or to an alternative gateway so that info is already there, it's just the process.

comments?

Back to Top
keizersozay View Drop Down
Groupie
Groupie
Avatar

Joined: 26 January 2005
Location: United States
Status: Offline
Points: 77
Post Options Post Options   Thanks (0) Thanks(0)   Quote keizersozay Quote  Post ReplyReply Direct Link To This Post Posted: 29 September 2005 at 8:29am

I'm not sure if I understand you but let me take a shot. You are suggesting that since SpamFilter sends email to our email server it can learn which email addresses are real since the email server (exchange or whatever) will either accept it or not. Then at night (or whenever) SpamFilter can go through and figure out which email addresses are actually active and only accept email for those?

If that is the case I don't think it will work for a few reasons. I and others here use another filter (Trend IMSS) behind SpamFilter for various reasons, so before the email gets to our email server it passes though another filter and SpamFilter never gets a response from the email server about the validity of the email address. Also, it would have to be a real time solution because when we add new email addresses, those people need to be able to get email immediately. If I told them it would take 24 hours or 'till the next day I will be flipping burgers before long.

With that being said I do like you idea. I had suggested in the past that SpamFilter have an optional LDAP connection to a server you specify (a domain controller) and for every email it could do a small query to see if the email address is real. This was shot down because it gets away from the main duty of SpamFilter and it was suggested that I script all valid email addresses out of our environment and have them automatically added to our approved sender list. This would work, but I haven't figured out how to do it yet.

Again, I’m not even sure if I understand your question correctly, so if I got it wrong just ignore all this.

Thanks

Back to Top
Kim View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote Kim Quote  Post ReplyReply Direct Link To This Post Posted: 29 September 2005 at 8:45am

What is Trend IMSS used for?

/Kim (got interested....)

Back to Top
keizersozay View Drop Down
Groupie
Groupie
Avatar

Joined: 26 January 2005
Location: United States
Status: Offline
Points: 77
Post Options Post Options   Thanks (0) Thanks(0)   Quote keizersozay Quote  Post ReplyReply Direct Link To This Post Posted: 29 September 2005 at 8:49am

The Trend Interscan Messaging Security Suite is used for several things. It does have some spam filter ability but that isn’t really too good unless you buy the addition spam piece and SpamFilter beats it hands down so we just use it for creating disclaimers and some other policy type rules. It also does all our virus checking, which works very well.

Back to Top
kspare View Drop Down
Senior Member
Senior Member


Joined: 26 January 2005
Location: Canada
Status: Offline
Points: 334
Post Options Post Options   Thanks (0) Thanks(0)   Quote kspare Quote  Post ReplyReply Direct Link To This Post Posted: 29 September 2005 at 9:38am

You kinda missed what i was getting at.

Midnight maint would do this, it would run through the database and collect email addy's.

For each email addy it would attempt to send an email to that address, when you connect to an smtp server, after helo the recipient data comes through, and at this point the email server will accept or decline the message. If the users smtp server declines the message it could be flagged for deletion.

Does that make a little more sense?

Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 29 September 2005 at 10:08am

Would be just easier to write a script to get the userlist for everyone and put it into the AuthorizedTo list on a regular basis. We poll our 2 mailservers every 10 minutes and download a list of all users (about 6200) and put them into the authorizedTo list. We poll this frequently so as new customers come on board and start adding email users they are automatically added in 10 minutes. Amazing how much stuff is stopped by the AuthorizedTo list.

Most mail servers should allow some type of scripting to get this info. We use Merak and Imail. Even if you use ldap you should be able to query the mailserver and build a list of valid email addresses and overwrite the AuthorizedTo list on a regular basis.

http://www.webguyz.net
Back to Top
keizersozay View Drop Down
Groupie
Groupie
Avatar

Joined: 26 January 2005
Location: United States
Status: Offline
Points: 77
Post Options Post Options   Thanks (0) Thanks(0)   Quote keizersozay Quote  Post ReplyReply Direct Link To This Post Posted: 29 September 2005 at 10:12am

can anyone offer some help on scripting valid email addresses out of an exchange/ldap environment?

Thanks.

Back to Top
kspare View Drop Down
Senior Member
Senior Member


Joined: 26 January 2005
Location: Canada
Status: Offline
Points: 334
Post Options Post Options   Thanks (0) Thanks(0)   Quote kspare Quote  Post ReplyReply Direct Link To This Post Posted: 29 September 2005 at 10:13am
No, not all of the mail the comes in is destined for our mail server, we provide a spam relay service for many customers so we don't always have direct access to their servers.
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 29 September 2005 at 10:24am

We have 3 customers who have Exchange servers and we provide relay services for. We wrote a web interface that allows their admin to enter their users login/pasword info into a table and have a script that polls that info every 10 minutes (same script as mentioned above) and also adds that to the AuthorizedTo list. This is helpful to us because we charge per user per month. Customers don't bug us every time they want to add or remove a user, they just update it themselves and in 10 minutes its active. We run a report of these users at the end of the month and bill accordingly.

For those who relay for others, do you just charge a flat fee per month??

http://www.webguyz.net
Back to Top
kspare View Drop Down
Senior Member
Senior Member


Joined: 26 January 2005
Location: Canada
Status: Offline
Points: 334
Post Options Post Options   Thanks (0) Thanks(0)   Quote kspare Quote  Post ReplyReply Direct Link To This Post Posted: 29 September 2005 at 10:29am

Well for now all the customers are also customers whom I manage their exchange server so it's not a big deal for billing, however that is an interesting way to look after billing! Currently we do email based entirely on domains, which has been successful, it is based on the honor system as far as billing goes.

Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 29 September 2005 at 10:50am

Another added benefit of doing this is that these exchange users can now log into the quarantine db with their own login/password to manage their own filtered mail. Most know to check once a day and we keep it for 48 hours.

We also have these exchange users use us as a smarthost so that we could get a list of everyones outgoing emails (as well as our local users) . We add to the AutoWhitelistDelivery.txt any emails that our valid customers send out. We do this with scripts we wrote to poll our dedicated outgoing mail server every 10 minutes and create valid SENDER|RECEIVER pairs for all emails that are sent out. The premise being that if they are sending an email they must be wanting a reply back.

Our AutoWhitelistDelivery.txt file is 1.3 meg big right now and has about 28,000 entries. This has cut down false positives down dramatically!!

 

http://www.webguyz.net
Back to Top
Alan View Drop Down
Groupie
Groupie


Joined: 06 May 2005
Location: United States
Status: Offline
Points: 43
Post Options Post Options   Thanks (0) Thanks(0)   Quote Alan Quote  Post ReplyReply Direct Link To This Post Posted: 26 July 2006 at 2:18pm
Hey WebGuyz,
I am a bit disappointed that the new LDAP feature in SF in not used for incoming.  But I see that you have another method to do something similar.  Could you provide some assitance or script to help with your method of automatically extracting LDAP emails from Exchange.  I am being hit hart by a spammer spoofing my domain with literally tens of  thousands of bounces just over the past two days.  You can contact me via the forum private messaging if you want to keep confidential.
Thanks for any help you can provide.
Back to Top
ImInAfrica View Drop Down
Groupie
Groupie
Avatar

Joined: 27 June 2006
Location: FL, USA
Status: Offline
Points: 60
Post Options Post Options   Thanks (0) Thanks(0)   Quote ImInAfrica Quote  Post ReplyReply Direct Link To This Post Posted: 30 July 2006 at 4:34pm
> can anyone offer some help on scripting valid email addresses out of an exchange/ldap environment?

What if SF builds a list out of rejected email addresses and caches those that were rejected for specified period of time as blacklisted?

Example:
- SF recieves an email for user1@domain.com
- SF accepts it and tries to forward it to the specified server.
- The recieving server rejects with "No User found".
- SF then caches the email user1@domain.com
- Further email to user1@domain.com are rejected for 24 hours (or 2 hours, or whatever is specified in the config file).

Pretty similar to what milter-ahead does in the *nix world.

Best
Amir
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 30 July 2006 at 10:14pm

Originally posted by Alan Alan wrote:

Hey WebGuyz,
I am a bit disappointed that the new LDAP feature in SF in not used for incoming.  But I see that you have another method to do something similar.  Could you provide some assitance or script to help with your method of automatically extracting LDAP emails from Exchange.  I am being hit hart by a spammer spoofing my domain with literally tens of  thousands of bounces just over the past two days.  You can contact me via the forum private messaging if you want to keep confidential.
Thanks for any help you can provide.

 

Alan,

  Take a look at this ADSI script below. If you can get it to work then maybe it can be modified to get a listing of Exchange mailboxes. I don't have Exchange but if its in the ballpark then maybe it can be a starting point. I can help you with the ASP, but not the Exchange testing. You may have to run this on the Exchange server.(hopefully you have to have IIS on that server)

http://support.microsoft.com/?kbid=241474

http://www.webguyz.net
Back to Top
Alan View Drop Down
Groupie
Groupie


Joined: 06 May 2005
Location: United States
Status: Offline
Points: 43
Post Options Post Options   Thanks (0) Thanks(0)   Quote Alan Quote  Post ReplyReply Direct Link To This Post Posted: 01 August 2006 at 4:53pm
Thanks.  I was able to find a solution for addresses via LDAP on Ex55/NT4.
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 02 August 2006 at 1:01pm
As an FYI, we are evaluating whether to allow LDAP/Active Directory checks for incoming emails, to validate existing recipients.

The issue is that doing so could cause a Denial of Service attack on the LDAP/AD servers, as massive emails can result in massive amounts of queries to the directory servers, potentially bringing them down.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
WebGuyz View Drop Down
Senior Member
Senior Member


Joined: 09 May 2005
Location: United States
Status: Offline
Points: 348
Post Options Post Options   Thanks (0) Thanks(0)   Quote WebGuyz Quote  Post ReplyReply Direct Link To This Post Posted: 02 August 2006 at 1:20pm

The blacklist cache would stop repeat dictionary attackers and if you cached users who have already been LDAP authorised (and held them in memory cache for a settable time) then the load would minimal on the LDAP (or whatever) server.

http://www.webguyz.net
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.262 seconds.