handling large number of domains |
Post Reply |
Author | |
atifghaffar
Senior Member Joined: 31 May 2006 Location: Switzerland Status: Offline Points: 104 |
Post Options
Thanks(0)
Posted: 31 May 2006 at 11:01pm |
Hello, I have installed the spamfilter isp (latest version) or 2 machines. I will use one as the incoming filter and the second as the outgoing filter.
The email setup is something like this. Email from the internet -> spamfilter-incoming -> incoming-mailserver -> if mail for internal users -> mailstore server else if mail for external domain -> spamfilter-outgoing ->outgoing-mailserver
Mails origionated internally go through the spamfilter-outgoing.
The incoming spamfilter is the one that will be connected to the internet and I have to define around 20,000 domains in it. When I tried to paste that in the interface the application hung. Question: How do you handle this large lists? Is there a way to make a lookup in a database or ldap or external script to know if the relay should be allowed for a certain domain? We expect to grow very fast to more domains so I would really like to know how other ISPs in the spam filter community handling this. thanks and best regards. |
|
LogSat
Admin Group Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
Post Options
Thanks(0)
|
atifghaffar,
SpamFilter should be able to handle that amount of domains with adequate hardware. Can you please let us know the amount of RAM and the type of CPU(s) on the server? Also, if you can please zip and email us at support@logsat.com the list of domains, we'll try to see if the list itself has any issues that could cause this. In the meantime, you could try to simply edit the "AllowedDomains.txt" file which contains the list of domains with a text editor, and paste the domains there. If SpamFilter is running, within 60 seconds it should automatically reload the list. If you wish, you can also stop SpamFilter, edit the file, and then restart it. |
|
atifghaffar
Senior Member Joined: 31 May 2006 Location: Switzerland Status: Offline Points: 104 |
Post Options
Thanks(0)
|
Hello Mr. Franceschetti , Thanks for your quick response. The hardware is a multi cpu x86 machine running vmware gsxserver. The spamfilter is running in the guest host with 1GB of ram. I can increase this to 2 or 3 GB if needed. I will try with editing the file directly. This is what we will be doing anyway with some script as we add domains. PS: Is there any possibility in the future to add a feature to make a lookup from a db?
|
|
best regards
Atif |
|
atifghaffar
Senior Member Joined: 31 May 2006 Location: Switzerland Status: Offline Points: 104 |
Post Options
Thanks(0)
|
I tried copying the file directly and the same behaviour. The application hung. Killing the service and trying to start the service gives the same behaviour. I am sending you the list of domains as a zip file for inspection. |
|
best regards
Atif |
|
Desperado
Senior Member Joined: 27 January 2005 Location: United States Status: Offline Points: 1143 |
Post Options
Thanks(0)
|
atif, There have been several threads about reading several of the lists out of a database and what we do ( and several other users) is to keep the domains, filters, etc in a database and use a stored procedure to create the changes in the text files. This works very well for us and the DB is linked with our customer/billing database so that a customer that cancels or a domain that expires doesn't stay in the spamfilter OR our Sendmail server. |
|
The Desperado
Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
|
vijay
Guest Group |
Post Options
Thanks(0)
|
hai, my name is vijay from hyd. here is problem for me. i con't configure the send mail on bugzill.I am getting 10053 error. i don't understand whether it's software prob's or my smtp server probs... what shall i do now... any body is there to help me.. regards & thanks.... vijay |
|
LogSat
Admin Group Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
Post Options
Thanks(0)
|
vijay,
This forum is dedicated to SpamFilter ISP, please keep all questions within the topic. |
|
LogSat
Admin Group Joined: 25 January 2005 Location: United States Status: Offline Points: 4104 |
Post Options
Thanks(0)
|
atif,
We did not receive the zip file yet. When you send it, could you also please include the AllowedDomainFilterMatrix.txt file as well? 1GB may be not enough, depending on the amount of emails you receive. With a 1-2GHz CPU and a gig of RAM you should be able to handle a few hundred thousands of emails per day. SpamFilter does not "care" about the number of domains, but rather it is the amount of emails that are received per day, and the number of concurrent connections at any time, that determine CPU and RAM usage. There may be a larger delay when inserting the domains for the first time, as some internal processing is required to arrange the list, but after this is done once, restarting SpamFilter will be much faster. With that amout of domains, depending on the amount of CPU that VMWare is assigning your virtual machine, it may be a minute or two the very first time this is done. Have you tried to wait 2-3 minutes to see if the list is processed? |
|
atifghaffar
Senior Member Joined: 31 May 2006 Location: Switzerland Status: Offline Points: 104 |
Post Options
Thanks(0)
|
Hello Mr. Franceschetti , I have sent a few emails to support@logsat.com which you never recieved I guess. Are you blocking my servers :-)?? I will try to send you the same file from a gmail account. PS: Would it be possible to make some changes to the software? Customization work?
best regards |
|
best regards
Atif |
|
atifghaffar
Senior Member Joined: 31 May 2006 Location: Switzerland Status: Offline Points: 104 |
Post Options
Thanks(0)
|
Doh. Mea Culpa. I am sorry, I have been sending sales/support mails to logstat.com instead of logsat.com. Forwarding now to support@logsat.com.
|
|
best regards
Atif |
|
atifghaffar
Senior Member Joined: 31 May 2006 Location: Switzerland Status: Offline Points: 104 |
Post Options
Thanks(0)
|
Hello all,
Thanks for your comments. Writting the AllowedDomains.txt works fine. I just needed to let it a couple of minutes to load the data into memory and update the AllowedDomainMatrix.txt file. Now the 2 spamfilter isp softwares are running on dedicated servers (no more vmware) with 4GB RAM each and are performing quiet well under really crazy load. I am generating the AllowedDomains, AllowedDomainMatrix and AuthorizedRecipients files with a process and then scp them to the server which runs the spamfilter incoming. Thanks for the tip Roberto to generate the files alphabetically sorted. Other than a couple of application crash I have not seen any serious problem. The AutoWhiteListForcedDelivery works very good too. |
|
best regards
Atif |
|
Post Reply | |
Tweet
|
Forum Jump | Forum Permissions You cannot post new topics in this forum You cannot reply to topics in this forum You cannot delete your posts in this forum You cannot edit your posts in this forum You cannot create polls in this forum You cannot vote in polls in this forum |
This page was generated in 0.164 seconds.