Not running a few hundred - but...

Running about 50 domains with SFE
~3000 users
~400000 emails per day
~25-40 concurrent connections
MSSQL 2000 - Win2003 Server
Running on a single Quad Core Xeon @2.4Ghz with 4GB RAM
mem and Virt mem are both ~850MB
CPU usage usually less than 10%

PM me if you 'd like more info.

jerbo128

 

Note that it takes about 3 minutes to load 300 domains. Probably take 10-13 or more for that many domains. Have a powerful box for that many domains. Have your db on a separate server

Can't believe you had that many on SFI. How big was your autowhitelistforcedelivery.txt file?

Other than that its been working just great. We have 2 SFE's to split the load and for redunadancy. SFE has surpassed all my expections with the excellent support of Roberto. I ran into a few minor issues initally but he fixed them all and now everything is great.

We created scripts to update our tblAuthorizedTo list and also have scripts to update the Autowhitelistforcedelivery with entries from our outbound mail server. Totally automated! The ability to use SQL code to update everything instead of text files is truly wonderful.

One nice thing that you can do is copy your existing spamfilter directory into a new directory and bring up the exe. There you can go thru the upgrade procedure while your production SFI is still cranking away. You can make sure everything is 100% before you shutdown your production server and convert it over.

Roberto,
   Maybe you should sticky the instructions for a painless upgrade like the ones you sent me which worked extremely well.

Hello Webguyz,

I was about to post a detailed reply, but these damn mac osx keyboard
shooot-cuts came across my way (not the same as kde/win), for example
Apple+Home took me to apple.com... welll whatever...

So here is some more information again.
We have a couple of machines running 2 instances of SPI (Okayed by
Roberto in terms of licensing).

Instance1 (incoming emails (mx and smtp-authed))
Instance 2 (outoging emails (this one can use a lot of help))

There is a lot in between, nothing very special though:
you can get an idea at (perhaps outdated when u read this) this diagram:
http://www.worldsoft-postmaster.info/


For the incoming spamfilter(the internet facing mx and smtpauth
machine): this is what I have from one of our machines
xyz # cat AllowedDomainFilterMatrix.txt | wc -l
20362

This is the amount of domains that SFI is handling. This file is updated
whenever we get/lose a domain.

This was the simple task, but we do not allow anything at anydomain
(obvioulsy), so we use the AuthorizedRecipients.

xyz: # cat AuthorizedRecipients.txt | wc -l
45093


The quarantine database is ofcoarse on a different machine running linux
+mysql (innodb)

There is a mailadmin machine that makes these different Authorized
maps and scp's(cygwin) them to the spamfilter-isp machines.

Hope this helps

and

looking forward to moving to SFE soon to avoid this hastle.

atifghaffar,

   I was just wondering how big some of your files must have been getting with that much data.

  Like yourself we also use SpamAssassin after our primary SFE's feeding into a second instance on one of the SFE's that checks for specific keywords and quarantines the email if found. It would have been nice to have SA incorporated inside of SFE, but in retrospect it makes more sense to use it as a final filter after SFE has done all the heavy lifting since SA is so very CPU intensive. Having SA check on each email would really require a lot of horsepower.

  I think you'll like SFE, but make sure the db is solid.

Edited by WebGuyz

Roerto,

Another glitch that I saw when moving to SFE was that the SFE expects the
SFE database to be on the same place as the quarantine database.

I hope, I am wrong.

The quarantine database needs a very poweful innodb based machine
while the SFE configuration can come from another moderate or a replica
machine.

Please correct me if I am wrong, I could not find where to separate these
2 databases.

Atif,

Both SFI and SFE share the same database. The SFI and SFE tables are completely independent, so there is no interaction between the data they contain, but they must reside in the same schema. This was done because in the future SpamFilter may add interaction between the tables, and also to simplify the design of future web interfaces to manage SFE (we're beta testing our soon-to-be-released one).

Please note that in the same database, if you're using MySQL, you can have some tables use the InnoDB engine and others using the MyISAM engine, so you can mix and match them as you please.

Yes, you may indeed want to differentiate the backups for the two sets of tables.

As a side-note, WebGuyz is absolutely correct when stating that if the SFE tables contain *lots* of data, the startup time will increase. It is also to note however, that the examples him and I were referring to assumed that each of the hundreds of domains was customized to have unique settings.

If you use SFE, but have default settings for the most part of the domains, the startup time will decrease dramatically. For example, while for starting up SFE with 200 domains, each customized, takes about 60 seconds on the 2GHz laptop, starting up with 600 domains, all inheriting the default settings, takes only about 10 seconds on the same laptop.