Spam Filter ISP Support Forum

  New Posts New Posts RSS Feed - limiting queue runners
  FAQ FAQ  Forum Search   Register Register  Login Login

limiting queue runners

 Post Reply Post Reply
Author
AndrewD View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote AndrewD Quote  Post ReplyReply Direct Link To This Post Topic: limiting queue runners
    Posted: 10 July 2006 at 8:04pm
Hi All,

I am currently using the trial version.  I have noticed that the queue runner is exceptionally agressive... Opening up 5+ threads to the same mailhost within a second.  Is there any way to stop this? (ideally cut down the number of queue runners, or limit the total number?)


Thanks,
Andrew
Back to Top
lyndonje View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 January 2006
Location: United Kingdom
Status: Offline
Points: 192
Post Options Post Options   Thanks (0) Thanks(0)   Quote lyndonje Quote  Post ReplyReply Direct Link To This Post Posted: 11 July 2006 at 3:46am
By queue runners do you mean TCP connections to your destination server?

I don't think you can limit/set this in SF, but I'd have thought you should be able to do this on your receiving server - ie set the maximum number of listening threads. If they're all in use, SF will queue the emails it couldn't send through and retry after n minutes set under 'Process queue every n minutes' under Settings/Configuration.
Back to Top
AndrewD View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote AndrewD Quote  Post ReplyReply Direct Link To This Post Posted: 11 July 2006 at 8:17pm
Hi,

Yes, this is what I meant... It is possible to set this in every other MTA that I have used :( - Oh well, I guess i'll just have to look at other software.

Thanks for your time :)

Cheers,
A

Originally posted by lyndonje lyndonje wrote:

By queue runners do you mean TCP connections to your destination server?

I don't think you can limit/set this in SF, but I'd have thought you should be able to do this on your receiving server - ie set the maximum number of listening threads. If they're all in use, SF will queue the emails it couldn't send through and retry after n minutes set under 'Process queue every n minutes' under Settings/Configuration.
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 11 July 2006 at 8:57pm
AndrewD,

SpamFilter has a limit on the maximum number of concurrent incoming connections (on the Settings - Configuration tab).

When it comes to forwarding queued emails to your destination SMTP server, SpamFilter will use one half of that value to calculate the number of maximum concurrent connections to your server.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
AndrewD View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote AndrewD Quote  Post ReplyReply Direct Link To This Post Posted: 12 July 2006 at 12:28am
Hi Roberto,

Thank you for your reply.  Is there any way to manually change the outgoing (forwarding) SMTP number of threads without touching the incoming limit? - I need to leave the incoming 40+, but I want to keep the outgoing to less than 5.

I see no real need to have so many queue runners hitting a single remote server at a time (within one second)...  Ideally the first queue runner should be spawned, and then another x (ie 5 mins later) - and so forth to deal with the load.  This is a very effective method for mail delivery (don't forget that multiple emails can be sent down one established SMTP session). 

If this is not possible for the current version, would you consider allowing this configuration to be changed in future versions?

Thanks again for your time.

Cheers,
Andrew.

Originally posted by LogSat LogSat wrote:

AndrewD,

SpamFilter has a limit on the maximum number of concurrent incoming connections (on the Settings - Configuration tab).

When it comes to forwarding queued emails to your destination SMTP server, SpamFilter will use one half of that value to calculate the number of maximum concurrent connections to your server.
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 12 July 2006 at 12:48am
AndrewD,

SpamFilter will forward any "clean" email it receives immediately in real-timeas it arrives. So assuming a listserver is sending massive legitimate emails to your users, and the incoming limit in SpamFilter is 40, in theory SpamFilter could have 40 concurrent incoming connections from that listserver. in this case, as emails are forwarded in real-time, SpamFilter will also have 40 outgoing concurrent connections to your destination mail server.

The only time when emails will be queued up is if your destination SMTP server is offline. In that case SpamFilter will queue the clean emails, and will attempt delivery on regular intervals. When your server will come back online, then in this case SpamFilter will forward all the queued emails to your destination SMTP server, using a number of concurrent threads equal to one half of the "Max incoming connection".

SpamFilter should forward as many emails as it can when flushing the queue, as when this happens it will be because your server was down. If it was down for several hours, there may potentially be hundred of thousands of clean emails queued for delivery, and they need to be processed as fast as possible to avoid further delays to end users.

You should size the value of "Max incoming connections" to a value that your destination SMTP server can handle.
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
lyndonje View Drop Down
Senior Member
Senior Member
Avatar

Joined: 31 January 2006
Location: United Kingdom
Status: Offline
Points: 192
Post Options Post Options   Thanks (0) Thanks(0)   Quote lyndonje Quote  Post ReplyReply Direct Link To This Post Posted: 12 July 2006 at 3:37am
Hi Andrew,

If SF was hadling 40 incomming connections, but only passing out 5, assuming your inbound feed was constant you'd end up with a bottleneck on SF. Why would you want that?

Also, if SF is receiving 40 incomming connections, what would happen if you didn't have SF? Your destination server would only have to process the 40 concurrent connections anyway, plus processing the spam. If you inbound load is too high for your destination server to handle, you need to upgrade or introduce a second for load balancing.

If you can limit the number of inbound connections on your destination server, which I assume by your earlier comment "It is possible to set this in every other MTA that I have used" that you can, why don't you do this? That way you'll have a bottleneck on your SF, which would be the same as if what you wanted to achieve was possible.
Back to Top
AndrewD View Drop Down
Guest Group
Guest Group
Post Options Post Options   Thanks (0) Thanks(0)   Quote AndrewD Quote  Post ReplyReply Direct Link To This Post Posted: 13 July 2006 at 8:48pm
Hi Again,

The reason for reducing threads is that the SMTP standard supports multiple smtp transactions in a single TCP connection (one after the after).  This is what the RFC dictates, and every other good MTA I have used follows these rules.

Typically you will have connections from all over the world (slow) being forwarded to a few internal severs. You also have the luxury of queuing internally (for a few seconds if needed).  It makes sense to follow the standards and not just spew TCP connections through, but instead group them by destination server and pass through seqentially (with additional queue runners if needed).

If you take this approach (and follow the standards) there is no bottle-neck, and things actually run much more efficiently on both ends).  I have worked on some exceptionally high volume mail systems (in the millions/day) - and this is the only good way to approach the situation.

I am not trying to be overly critical of this product, instead pointing out a weaknesses that could be improved upon.  I just can't see your product being used in a high volume isp environment.

Thanks,
Andrew.

Originally posted by lyndonje lyndonje wrote:

Hi Andrew,

If SF was hadling 40 incomming connections, but only passing out 5, assuming your inbound feed was constant you'd end up with a bottleneck on SF. Why would you want that?

Also, if SF is receiving 40 incomming connections, what would happen if you didn't have SF? Your destination server would only have to process the 40 concurrent connections anyway, plus processing the spam. If you inbound load is too high for your destination server to handle, you need to upgrade or introduce a second for load balancing.

If you can limit the number of inbound connections on your destination server, which I assume by your earlier comment "It is possible to set this in every other MTA that I have used" that you can, why don't you do this? That way you'll have a bottleneck on your SF, which would be the same as if what you wanted to achieve was possible.
Back to Top
LogSat View Drop Down
Admin Group
Admin Group
Avatar

Joined: 25 January 2005
Location: United States
Status: Offline
Points: 4104
Post Options Post Options   Thanks (0) Thanks(0)   Quote LogSat Quote  Post ReplyReply Direct Link To This Post Posted: 13 July 2006 at 11:29pm
Andrew,

SpamFilter is not strictly an MTA. It is a product that has to receive emails, completely reject some of them, archive others, and pass on to other servers clean ones. To do so efficiently, unlike what you state, the best way for us to do so is to pass on each email in the same way it was received. This means that each incoming connection will result in a outgoing connection if the email is clean.

Performance-wise, please post a new query in this forum asking for other user's experience. This will allow you not to take our word when we say that a very small 3MB application can indeed handle millions of emails/day, on very simple hardware (as in a VMWare guest machine with low CPU and RAM available).
Roberto Franceschetti

LogSat Software

Spam Filter ISP
Back to Top
 Post Reply Post Reply
  Share Topic   

Forum Jump Forum Permissions View Drop Down



This page was generated in 0.164 seconds.