Anyone using SFE with a few hundred domai
Printed From: LogSat Software
Category: Spam Filter ISP
Forum Name: Spam Filter ISP Support
Forum Description: General support for Spam Filter ISP
URL: https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=6107
Printed Date: 05 February 2025 at 5:51pm
Topic: Anyone using SFE with a few hundred domai
Posted By: WebGuyz
Subject: Anyone using SFE with a few hundred domai
Date Posted: 16 June 2007 at 11:39pm
Interested in hearing from anyone using SFE with a few hundred domains. Wanted to compare notes and see what your performance is like.
Thanks!
------------- http://www.webguyz.net
|
Replies:
Posted By: jerbo128
Date Posted: 17 June 2007 at 12:49pm
Not running a few hundred - but...
Running about 50 domains with SFE ~3000 users ~400000 emails per day ~25-40 concurrent connections MSSQL 2000 - Win2003 Server Running on a single Quad Core Xeon @2.4Ghz with 4GB RAM mem and Virt mem are both ~850MB CPU usage usually less than 10%
PM me if you 'd like more info.
jerbo128
|
Posted By: atifghaffar
Date Posted: 17 July 2007 at 9:45pm
WebGuys, I am running the SFI with 20K+ domains with the intention of
moving to SFE.
What should I be careful about?
------------- best regards
Atif
|
Posted By: WebGuyz
Date Posted: 17 July 2007 at 10:01pm
Note that it takes about 3 minutes to load 300 domains. Probably take 10-13 or more for that many domains. Have a powerful box for that many domains. Have your db on a separate server
Can't believe you had that many on SFI. How big was your autowhitelistforcedelivery.txt file?
Other than that its been working just great. We have 2 SFE's to split the load and for redunadancy. SFE has surpassed all my expections with the excellent support of Roberto. I ran into a few minor issues initally but he fixed them all and now everything is great.
We created scripts to update our tblAuthorizedTo list and also have scripts to update the Autowhitelistforcedelivery with entries from our outbound mail server. Totally automated! The ability to use SQL code to update everything instead of text files is truly wonderful.
One nice thing that you can do is copy your existing spamfilter directory into a new directory and bring up the exe. There you can go thru the upgrade procedure while your production SFI is still cranking away. You can make sure everything is 100% before you shutdown your production server and convert it over.
Roberto, Maybe you should sticky the instructions for a painless upgrade like the ones you sent me which worked extremely well.
------------- http://www.webguyz.net
|
Posted By: LogSat
Date Posted: 17 July 2007 at 10:25pm
Atif,
WebGuyz is correct, we can provide you with a rather simple set of instructions to install another instance of SpamFilter that can run in Enterprise mode, using your existing settings and database, but without interfering with your live install. This will allow you to perform any tests you wish to see if SFE will work for you. If you can contact us by email I'll provide them to you, and yes, we really should add them to the documentation and/or the forum :-)
For comparison, we just tested SFE on a laptop (the machine I have available right now) with a single Intel Core 2 Duo 2GHz CPU, running both SFE and SQL. SFE started up in just about 60 seconds with slightly over 200 domains.
------------- Roberto Franceschetti
http://www.logsat.com" rel="nofollow - LogSat Software
http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP
|
Posted By: atifghaffar
Date Posted: 17 July 2007 at 10:41pm
Hello Webguyz,
I was about to post a detailed reply, but these damn mac osx keyboard
shooot-cuts came across my way (not the same as kde/win), for example
Apple+Home took me to apple.com... welll whatever...
So here is some more information again.
We have a couple of machines running 2 instances of SPI (Okayed by
Roberto in terms of licensing).
Instance1 (incoming emails (mx and smtp-authed))
Instance 2 (outoging emails (this one can use a lot of help))
There is a lot in between, nothing very special though:
you can get an idea at (perhaps outdated when u read this) this diagram:
http://www.worldsoft-postmaster.info/
For the incoming spamfilter(the internet facing mx and smtpauth
machine): this is what I have from one of our machines
xyz # cat AllowedDomainFilterMatrix.txt | wc -l
20362
This is the amount of domains that SFI is handling. This file is updated
whenever we get/lose a domain.
This was the simple task, but we do not allow anything at anydomain
(obvioulsy), so we use the AuthorizedRecipients.
xyz: # cat AuthorizedRecipients.txt | wc -l
45093
The quarantine database is ofcoarse on a different machine running linux
+mysql (innodb)
There is a mailadmin machine that makes these different Authorized
maps and scp's(cygwin) them to the spamfilter-isp machines.
Hope this helps
and
looking forward to moving to SFE soon to avoid this hastle.
------------- best regards
Atif
|
Posted By: WebGuyz
Date Posted: 17 July 2007 at 10:56pm
LogSat wrote:
Atif,
For comparison, we just tested SFE on a laptop (the machine I have available right now) with a single Intel Core 2 Duo 2GHz CPU, running both SFE and SQL. SFE started up in just about 60 seconds with slightly over 200 domains.
|
Yeah, 60 seconds, but does each of the 200 domains have over 8,000 entries in the tblbl_keywords and do you have over 350k entries in your autowhitelist table.
I guess its all relative, but you will need some ponies with 20+K domains.
I love SFE and glad we 'talked' Roberto and his team into writing it.![](smileys/smiley2.gif)
------------- http://www.webguyz.net
|
Posted By: WebGuyz
Date Posted: 17 July 2007 at 11:06pm
atifghaffar,
I was just wondering how big some of your files must have been getting with that much data.
Like yourself we also use SpamAssassin after our primary SFE's feeding into a second instance on one of the SFE's that checks for specific keywords and quarantines the email if found. It would have been nice to have SA incorporated inside of SFE, but in retrospect it makes more sense to use it as a final filter after SFE has done all the heavy lifting since SA is so very CPU intensive. Having SA check on each email would really require a lot of horsepower.
I think you'll like SFE, but make sure the db is solid.
------------- http://www.webguyz.net
|
Posted By: atifghaffar
Date Posted: 17 July 2007 at 11:07pm
Roberto,
I am moving the current infrastructure from the shared-isp-infrastructure
to the mail-only-infrastructure and my first insinct was also to move to
SFE, but:
1. I want the move to be very smooth. just IP address changes at the MX
level.
2. Already separating smtpauth and MX (SF(I|E) is great for MX but as
smtpauth machine, I would prefer the basebones (syslogging) postfix
3. Fallback. In case, there is any problems, I would like to fallback on the
current machines (running untouched version of SFI, to be replaced as
soon as the migration is done)
When the 1 to 1 migration is done, I will have the previous set of
machines to play with SFE and to acquire more licences :-)
------------- best regards
Atif
|
Posted By: atifghaffar
Date Posted: 17 July 2007 at 11:16pm
Roerto,
Another glitch that I saw when moving to SFE was that the SFE expects the
SFE database to be on the same place as the quarantine database.
I hope, I am wrong.
The quarantine database needs a very poweful innodb based machine
while the SFE configuration can come from another moderate or a replica
machine.
Please correct me if I am wrong, I could not find where to separate these
2 databases.
------------- best regards
Atif
|
Posted By: WebGuyz
Date Posted: 17 July 2007 at 11:38pm
atifghaffar,
Same o Same o.
Thats why I said you needed a solid db. Doing the upgrade as Roberto described did not mess with my production SFI because there was no actual traffic going to the copied spamfilter directory that I was using to do the upgrade. Was a little hesitant about that and it took Roberto a couple of trys to convince me that I wouldn't screw my production system even though I was upgrading the same database my production system was using. A little unnerving at first but he was right. You can switch back and for between SFE and SFI and each system would be independant of the other. Pretty good design in that regard.
------------- http://www.webguyz.net
|
Posted By: LogSat
Date Posted: 18 July 2007 at 11:33am
Atif,
Both SFI and SFE share the same database. The SFI and SFE tables are completely independent, so there is no interaction between the data they contain, but they must reside in the same schema. This was done because in the future SpamFilter may add interaction between the tables, and also to simplify the design of future web interfaces to manage SFE (we're beta testing our soon-to-be-released one).
Please note that in the same database, if you're using MySQL, you can have some tables use the InnoDB engine and others using the MyISAM engine, so you can mix and match them as you please.
------------- Roberto Franceschetti
http://www.logsat.com" rel="nofollow - LogSat Software
http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP
|
Posted By: atifghaffar
Date Posted: 18 July 2007 at 2:36pm
Roberto, I understand your reasons and myisam/innodb is not the issue.
One example: Backup database without quarantine (just domains, prefs, etc) : 5MB
Backup database quarantine 50GB+
Perhaps we can exclude some tables from the backup...
------------- best regards
Atif
|
Posted By: LogSat
Date Posted: 18 July 2007 at 3:06pm
Yes, you may indeed want to differentiate the backups for the two sets of tables.
As a side-note, WebGuyz is absolutely correct when stating that if the SFE tables contain *lots* of data, the startup time will increase. It is also to note however, that the examples him and I were referring to assumed that each of the hundreds of domains was customized to have unique settings.
If you use SFE, but have default settings for the most part of the domains, the startup time will decrease dramatically. For example, while for starting up SFE with 200 domains, each customized, takes about 60 seconds on the 2GHz laptop, starting up with 600 domains, all inheriting the default settings, takes only about 10 seconds on the same laptop.
------------- Roberto Franceschetti
http://www.logsat.com" rel="nofollow - LogSat Software
http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP
|
Posted By: LogSat
Date Posted: 19 July 2007 at 7:30am
I just realized that we already have a detailed description of how to perform the upgrade to SpamFilter Enterprise offline in SpamFilter's new help file!
The full install of the current SpamFilter official release always includes the full manual, and the procedure is described in the chapter:
Upgrading to SpamFilter Enteprise
"offline"
...I guess I'm included in the very popular group of admins who doesn't read manuals...
I must admit that I didn't remember it being there either, but that page pooped up on a Google search I was performing, and that's how I then realized it was there ![](smileys/smiley1.gif)
We'll post the article to the forum as well soon.
------------- Roberto Franceschetti
http://www.logsat.com" rel="nofollow - LogSat Software
http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP
|
|