Print Page | Close Window

handling large number of domains

Printed From: LogSat Software
Category: Spam Filter ISP
Forum Name: Spam Filter ISP Support
Forum Description: General support for Spam Filter ISP
URL: https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=5633
Printed Date: 27 December 2024 at 1:53pm


Topic: handling large number of domains
Posted By: atifghaffar
Subject: handling large number of domains
Date Posted: 31 May 2006 at 11:01pm

Hello,

I have installed the spamfilter isp (latest version) or 2 machines.

I will use one as the incoming filter and the second as the outgoing filter.

 

The email setup is something like this.

Email from the internet -> spamfilter-incoming -> incoming-mailserver -> if mail for internal users -> mailstore server

else if mail for external domain -> spamfilter-outgoing ->outgoing-mailserver

 

Mails origionated internally go through the spamfilter-outgoing.


The outgoing part is working fine (This server can only be accessed from internal hosts). It has * as Local Domains

 

The incoming spamfilter is the one that will be connected to the internet and I have to define around 20,000 domains in it.

When I tried to paste that in the interface the application hung.

Question: How do you handle this large lists?

Is there a way to make a lookup in a database or ldap or external script to know if the relay should be allowed for a certain domain?

We expect to grow very fast to more domains so I would really like to know how other ISPs in the spam filter community handling this.

thanks and best regards.


Replies:
Posted By: LogSat
Date Posted: 31 May 2006 at 11:27pm
atifghaffar,

SpamFilter should be able to handle that amount of domains with adequate hardware. Can you please let us know the amount of RAM and the type of CPU(s) on the server?
Also, if you can please zip and email us at support@logsat.com the list of domains, we'll try to see if the list itself has any issues that could cause this.
In the meantime, you could try to simply edit the "AllowedDomains.txt" file which contains the list of domains with a text editor, and paste the domains there. If SpamFilter is running, within 60 seconds it should automatically reload the list. If you wish, you can also stop SpamFilter, edit the file, and then restart it.


-------------
Roberto Franceschetti

http://www.logsat.com" rel="nofollow - LogSat Software

http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP

Posted By: atifghaffar
Date Posted: 01 June 2006 at 1:04am

Hello Mr. Franceschetti ,

Thanks for your quick response.

The hardware is a multi cpu x86 machine running vmware gsxserver.

The spamfilter is running in the guest host with 1GB of ram. I can increase this to 2 or 3 GB if needed.

I will try with editing the file directly. This is what we will be doing anyway with some script as we add domains.

PS: Is there any possibility in the future to add a feature to make a lookup from a db?

 



-------------
best regards

Atif

Posted By: atifghaffar
Date Posted: 01 June 2006 at 1:26am

I tried copying the file directly and the same behaviour. The application hung.

Killing the service and trying to start the service gives the same behaviour.

I am sending you the list of domains as a zip file for inspection.



-------------
best regards

Atif

Posted By: Desperado
Date Posted: 01 June 2006 at 6:44am

atif,

There have been several threads about reading several of the lists out of a database and what we do ( and several other users) is to keep the domains, filters, etc in a database and use a stored procedure to create the changes in the text files.  This works very well for us and the DB is linked with our customer/billing database so that a customer that cancels or a domain that expires doesn't stay in the spamfilter OR our Sendmail server.



-------------
The Desperado
Dan Seligmann.
Work: http://www.mags.net
Personal: http://www.desperado.com

Posted By: Guests
Date Posted: 01 June 2006 at 8:02am

hai,

my name is vijay from hyd. here is problem for me. i con't configure the send mail on bugzill.I am getting 10053 error. i don't understand whether it's software prob's or my smtp server probs...

what shall i do now...

any body is there to help me..

regards & thanks....

vijay

Posted By: LogSat
Date Posted: 01 June 2006 at 4:38pm
vijay,

This forum is dedicated to SpamFilter ISP, please keep all questions within the topic.


-------------
Roberto Franceschetti

http://www.logsat.com" rel="nofollow - LogSat Software

http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP

Posted By: LogSat
Date Posted: 01 June 2006 at 4:47pm
atif,

We did not receive the zip file yet. When you send it, could you also please include the AllowedDomainFilterMatrix.txt file as well?

1GB may be not enough, depending on the amount of emails you receive. With a 1-2GHz CPU and a gig of RAM you should be able to handle a few hundred thousands of emails per day. SpamFilter does not "care" about the number of domains, but rather it is the amount of emails that are received per day, and the number of concurrent connections at any time, that determine CPU and RAM usage.

There may be a larger delay when inserting the domains for the first time, as some internal processing is required to arrange the list, but after this is done once, restarting SpamFilter will be much faster. With that amout of domains, depending on the amount of CPU that VMWare is assigning your virtual machine, it may be a minute or two the very first time this is done. Have you tried to wait 2-3 minutes to see if the list is processed?


-------------
Roberto Franceschetti

http://www.logsat.com" rel="nofollow - LogSat Software

http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP

Posted By: atifghaffar
Date Posted: 08 June 2006 at 7:59am

Hello Mr. Franceschetti ,

I have sent a few emails to mailto:support@logsat.com - support@logsat.com which you never recieved I guess.  Are you blocking my servers :-)??

I will try to send you the same file from a gmail account.

PS: Would it be possible to make some changes to the software? Customization work?

 

best regards



-------------
best regards

Atif

Posted By: atifghaffar
Date Posted: 08 June 2006 at 8:09am

Doh. Mea Culpa.

I am sorry, I have been sending sales/support mails to logstat.com instead of logsat.com.

Forwarding now to mailto:support@logsat.com - support@logsat.com .

 

 

 



-------------
best regards

Atif

Posted By: atifghaffar
Date Posted: 14 June 2006 at 8:17pm
Hello all,

Thanks for your comments.
Writting the AllowedDomains.txt works fine. I just needed to let it a couple of minutes to load the data into memory and update the AllowedDomainMatrix.txt file.

Now the 2 spamfilter isp softwares are running on dedicated servers  (no more vmware) with 4GB RAM each and are performing quiet well under really crazy load.

I am generating the AllowedDomains, AllowedDomainMatrix and AuthorizedRecipients files with a process and then scp them to the server which runs the spamfilter incoming.

Thanks for the tip Roberto to generate the files alphabetically sorted.


Other than a couple of application crash I have not seen any serious problem.

The AutoWhiteListForcedDelivery works very good too.


-------------
best regards

Atif


Print Page | Close Window