Stuck Connections
Printed From: LogSat Software
Category: Spam Filter ISP
Forum Name: Spam Filter ISP Support
Forum Description: General support for Spam Filter ISP
URL: https://www.logsat.com/spamfilter/forums/forum_posts.asp?TID=5637
Printed Date: 05 February 2025 at 9:57am
Topic: Stuck Connections
Posted By: jerbo128
Subject: Stuck Connections
Date Posted: 05 June 2006 at 9:38pm
|
Starting about a week ago, I began to have "stuck connections" Anywere from 2 to 10 per day. Clicking on the KILL line will not clear them. They are not reflected in the "current inbound connections" along the status bar of the SF GUI. The only way that I can clear them is to restart the SF service. The screenshot below was taken at 8PM. You can see that there are several connections that are 6+ hours old. I pasted the log from the timeperiod of the last stuck connection. I see nothing that sticks out other than the fact that the log never showed the "disconnect". I am running 3.0.1.558. I have been running this version since the day it came out and have had no other issues at all. As always, thanks for the help. You have an outstanding product.
******************************* 06/05/06 14:37:14:792 -- (34704) Connection from: 200.123.153.9 - Originating country : Argentina ********************************** |
Replies:
Posted By: LogSat
Date Posted: 06 June 2006 at 12:13am
|
We've had 3 reports today with a similar issue.
It's something that has rarely come up before, and we believed was solved in
version 3.0.1.557. It's a strange coincidence to receive so
many reports the same day, but so far we've not been able to reproduce the
problem. As soon as we find soemthing more I'll let you
know.
------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: jerbo128
Date Posted: 06 June 2006 at 9:24am
|
Let me know if there is anything that you need - logs, screenshots, etc. Thanks Roberto. jerbo128 |
Posted By: LogSat
Date Posted: 06 June 2006 at 4:15pm
|
jerbo128, actually a zipped copy of your SpamFilter activity log for 6/5/06, may be trimmed from 11am to 2pm, could hopefully help us finding out why those connections still show up. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: jerbo128
Date Posted: 06 June 2006 at 5:31pm
|
Log sent to support@..... Thanks for the help. jerbo128 |
Posted By: lyndonje
Date Posted: 13 June 2006 at 7:50am
| For the record I noticed this again on my install. I hadn't checked it for a while, and there were stuck connections going back days (possible more than a week). I'm running 3.0.1.558 and as I wasn't aware of any cure just restarted the SF service. |
Posted By: mikek
Date Posted: 13 June 2006 at 9:24am
|
I'm seeing stuck connections as well - looks like they show up when load on spamfilter is heavy... connections are no longer existant on the system (netstat) but show up on the connections tab and can not be killed there |
Posted By: LogSat
Date Posted: 13 June 2006 at 11:15pm
|
We have recently found an issue with connections steadily increasing in the following specific case. 1. Using MySQL and the MyISAM database type (InnoDB is fine, it does not cause the problem). 2. The MySQL database is being backed up using the :Lock All Tables" execution method. 3. The database backup process takes several minutes to complete. Under the scenario above, MySQL will place a write lock on all tables, preventing any application from making DB updates. This will prevent SpamFilter to add new records. We have safety features that timeout the update process in case the SQL update take too long. However, due to a "missing feature" (i.e. a bug....) in the MySQL ODBC driver, the timeout function is not implemented correctly. This causes SpamFilter to wait indefinetly until the lock is removed. The connection thread thus will not terminate until the email has been inserted in the database, and this will cause the number of incoming connections thread to increase steadily, until the lock is lifted. But if the backup process takes several minutes to complete, this may cause the "Too many connections" limit to be reached. It is to be noted that the problem occurs only in the specific scenario above. If the MySQL database is unavailable for any other reason (DB is down, non-existent tables, incorrect permissions for example), SpamFilter will immediately see the DB connectivity issues and will simply continue processing emails without quarantining them. We've just uploaded build 3.0.1.571 that addresses this problem and prevents it from occurring. This did require major changes to how quarantined emails are processed, so, while testing has shown no issues, it is to note that this *is* a pre-release build, and as such, may not be as stable as an official release. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: lyndonje
Date Posted: 14 June 2006 at 3:54am
|
My problem must have been something else then as I'm using MS SQL 2005??? |
Posted By: jerbo128
Date Posted: 14 June 2006 at 8:38am
|
FYI: I am running Ms Access - and for the record have not had any more of these "stuck connections" in the past week. jerbo128 |
Posted By: mikek
Date Posted: 16 June 2006 at 5:35am
|
I'm still seeing these stuck connections even with the newest build .571 SQL Server 2000 via OLEDB as DB Server |
Posted By: lyndonje
Date Posted: 16 June 2006 at 10:14am
|
I'm seeing stuck connections again. ATM SF is running at 98% CPU. There
are 24 Stuck connections from yesterday between 22:19:52 and 22:22:33
BST (3 minute period). They are stuck in either QUIT, QUEUEING EMAIL or
RCPT TO. I have another 3 stuck from today, at 15:02:57, 11:16:40 & 11:26:57. In NOOP, and PROCESSING DATA... Even though none of the IPs show in a netstat. I've zipped and emailed an hours worth of logs for you from yesterday. Regards, Lyndon. |
Posted By: WebGuyz
Date Posted: 16 June 2006 at 1:09pm
|
I see anywhere from 5-12 stuck connections per day and have been restarting SFI every evening. ------------- http://www.webguyz.net |
Posted By: LogSat
Date Posted: 17 June 2006 at 11:43am
|
Thanks to everyone for their logs. Unfortunately as of today we still have not been able to reproduce the problem. We did however completely rewrite the procedure that checks for idle connections and diisconnects them. We're going in a bit blind as we don't know the cause, but we've tried to forsee as many scenarios as possible. A new pre-release build (3.0.1.573) is available in the registered user area to attempt one more shot at adressing this. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: lyndonje
Date Posted: 19 June 2006 at 4:15am
|
Have updated to the pre-release, will let you know what happens. Regards, Lyndon. |
Posted By: lyndonje
Date Posted: 20 June 2006 at 6:38am
| So far so good.... no stuck connections yet. |
Posted By: mikek
Date Posted: 20 June 2006 at 6:49am
| Can't find build .573 on the download page... .571 is the newest version listed, although I have .572 installed. |
Posted By: lyndonje
Date Posted: 20 June 2006 at 6:54am
|
I've just checked that I'm running 3.0.2.573 and I am. I've just logged into the registered area and can now only see: http:// - http:// - SF2.6.3.487.zip http:// - http:// - SF2.7.1.535.zip http:// - http:// - SF3.0.1.560.zip http:// - http:// - SF3.0.1.561.zip http:// - http:// - SF3.0.1.567.zip http:// - http:// - SF3.0.2.571.zip Roberto, where has it gone? Must have been there before for me to download...? |
Posted By: LogSat
Date Posted: 20 June 2006 at 8:21am
|
lyndonje, mikek, Yesterday we had 2 separate reports of SpamFilter freezing after just a few minutes. It affected both builds 572 and 573. While we were not able to replicate it, and there were only 2 reports, they were identical in the symptom... so that was still too many for us... We recalled those releases yesterday as a precaution. We've just uploaded build 575 which should have fixed those issues. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: WebGuyz
Date Posted: 20 June 2006 at 8:36am
|
My .573 had locked up once as well but I was going to see if it happened again. Too bad it happened at 4:00am and my monitoring system was able to page me and wake me up. Will load up 575 tonite and keep my fingers crossed. ------------- http://www.webguyz.net |
Posted By: lyndonje
Date Posted: 20 June 2006 at 8:48am
|
Funny that, just read this post and at the same time noticed SF had
locked up on me too! White screen, not responding and not listening on
port 25. Just loaded 575. Will keep you posted. |
Posted By: lyndonje
Date Posted: 20 June 2006 at 10:07am
|
Bad news, running 3.0.2.575 I have 6 stuck connections. Have emailed you further details. |
Posted By: mikek
Date Posted: 20 June 2006 at 11:09am
|
already 4 stuck connections here with .575... inform me, if you want logs... |
Posted By: lyndonje
Date Posted: 20 June 2006 at 11:17am
|
Hi mikek, out of interest do you use the 'Authorized To' filter? If so, are the stuck connections to recipients that are either covered by *@domain or not listed in your Authorised To file? I might be way off here, so don't want to cloud things or lead people down the wrong path, but thats what I have noticed on these recent stuck connections. |
Posted By: mikek
Date Posted: 20 June 2006 at 11:21am
| Yes, I do. I generate a authorizedto.txt file out of the user database of our mailserver, every time a change to that database happens. The file has about 4700 entries. |
Posted By: lyndonje
Date Posted: 20 June 2006 at 11:26am
|
You may not have seen my edited comment at 4:19, I added: If so, are the stuck connections to recipients that are either covered by *@domain or not listed in your Authorised To file? I might be way off here, so don't want to cloud things or lead people down the wrong path, but thats what I have noticed on these recent stuck connections. |
Posted By: LogSat
Date Posted: 20 June 2006 at 11:31am
|
thanks for pointing that out, we'll look into that aspect right now. As we're still not able to replicate this, we're open to any suggestions / hints you all have.
------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: mikek
Date Posted: 20 June 2006 at 11:33am
| No, all my recent stuck connections are to users which are listed individually in the authorized-to list. |
Posted By: lyndonje
Date Posted: 20 June 2006 at 11:35am
Oh right... thats that one out the window then  Oh well, worth a try.
|
Posted By: mikek
Date Posted: 21 June 2006 at 8:46am
| 18 stuck connections in the last 24 hours. sorry to say that looks to me like it even got worse with .575 |
Posted By: lyndonje
Date Posted: 21 June 2006 at 9:13am
| You could accuse me of thinking the same - first stuck connection 20/06 @ 16:26, since then I have 19 in total. |
Posted By: Desperado
Date Posted: 21 June 2006 at 7:49pm
|
All, I have been following this "Stuck Connection" thread for a while and am wondering under what conditions they are showing up? I have a fairly high traffic site (250,00 messages a day) and see no evidence of this at all. ------------- The Desperado Dan Seligmann. Work: http://www.mags.net Personal: http://www.desperado.com |
Posted By: lyndonje
Date Posted: 23 June 2006 at 5:59am
|
Hello all... Something else for you to try, not sure if this has anything to do with the stuck connections, but worth a try. The other day I decided for no reason at all to uncheck the "Enable Activity Log" under the Activity Log tab. Since then I don't seem to of had any stuck connections? Again this may be totally unrelated but thought I'd let you know so you can try it (if you don't already have the activity log disabled) to see if it makes any difference. Regards, Lyndon. |
Posted By: jerbo128
Date Posted: 23 June 2006 at 9:47am
|
Our background: We are ruining spam filter on a Win2003 server that also runs MS DNS. We do not receive a large amount of email - about 250K per month. I leave the SF GUI running all the time, usually looking at the log screen. We quarantine to a Ms access database which usually runs around 100-150MB. Our quarantined emails expire after 7 days. I would have no problem emailing in any of our files. These stuck connections may not happen for 2-3 days, and then I may get 10 in one day. I do not have an automated "service restart". The service has never failed on us, however we usually do restart it once a week or so. If there's anything I can do to help at all, let me know. jerbo |
Posted By: LogSat
Date Posted: 23 June 2006 at 10:03am
|
Everyone, When a connection is disconnected, the following occurs, in order, one right after the other: 1. The "Current inbound connections" counter is decreased. 2. The IP is added to the blacklist cache if it is spam 3. The "Disconnect" log entry is added to the activity logfile 4. The connection is removed from the Connections tab. What we're seing in the logs we've received is that, for the connections that "appear" stuck, only events (1) and (2) happen. On the other hand, (3) and (4) are never executed, so the disconnect event is never logged to the logfile (and on the screen), and the connection is not removed from the list. Lyndon, you were right on the money on this one i believe, as I do think it is indeed tied to the activity log. We've narrowed this down a lot thanks to all your help. Now that we know what happens, we're trying to figure out "why" and how to prevent it. We may have a new build later today. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: LogSat
Date Posted: 23 June 2006 at 11:41am
|
Well.... build 3.0.1.578 is available to try one more time taking care of these stuck connections. If this works... I'll offer virtual beers for everyone!
------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: Dan B
Date Posted: 23 June 2006 at 8:13pm
|
I would say that the issue has been fixed with that .578 build. All my stuck connections and very high CPU usage has stopped. My servers are running back to normal. Thanks, Roberto for the quick fix. Dan |
Posted By: Dan B
Date Posted: 27 June 2006 at 12:32pm
|
R, The .578 build has help a lot. But I have question about the order of the settings within the ini file. We have 4 identical servers same everything. Spamserver 1 & 4 take care of our domain1.net equal MX preference of 5 and spamserver 2 & 3 take care of our domain housing domains. I noticed that server 4 never had a problem with the 100% CPU and stuck connections. Server 1,2,3 had those issues very bad even with the latest build. Just for the heck of it, I copied the ini file from spamserver 4 and placed it on spamserver 1 and just changed the ip address, qualified name and serverid. As soon as I restarted the service on spamserver 1 the stuck connections and the 100% CPU issues went away and didn’t come back. Since it did so well, I did the same to the other 2 servers. Exact same thing, all issues went away. Thanks, |
Posted By: LogSat
Date Posted: 27 June 2006 at 1:32pm
|
Everything is possible, especially since we were never able to replicate this behavior to begin with. I think we know "where" this was occurring in the code, but not "why". ...and without knowing the "why" it's very hard to fix the issue, as we do not know what the issue is...! Are the stuck connections still present in 578? If you still have the two ini files that exibit the two different behaviors, I'd like to compare them to see if there's any clues. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: Dan B
Date Posted: 27 June 2006 at 1:43pm
|
I have the all 4 before and after the changes. I will zip them up and send to you. Do you need any logs from those servers? Dan
|
Posted By: Dan B
Date Posted: 27 June 2006 at 3:25pm
|
R, Yes, the stuck connections were still present with 578 until I replaced the ini file with spamserver4's ini file. Dan
|
Posted By: lyndonje
Date Posted: 28 June 2006 at 4:01am
|
I don't think I've had anymore stuck connections since installing 3.0.2.578. |
Posted By: jerbo128
Date Posted: 28 June 2006 at 1:18pm
|
Today, I have noticed that I always have 2+ incoming connections (from the status bar) however, when I look at the connections screen, those first 2 are not shown. For example, when no connections show on the grid, the counter says 2. When I have 5 active connections on the grid, the counter shows 7. Any way that plays into your current thoughts with this stuck connections issue? I am running the 3.0.2.578. jerbo128 |
Posted By: WebGuyz
Date Posted: 28 June 2006 at 4:32pm
I just checked my connection and had 4 showing but the incoming was showing about 14, 10 more then the actual connections screen was showing. As the number of connections increased/decreased the incoming was always showing 10 more than the conections screen. Its like the connections are still getting stuck we just just can't see them.  running .578------------- http://www.webguyz.net |
Posted By: lyndonje
Date Posted: 29 June 2006 at 3:36am
| Mine seems to be +9 |
Posted By: mikek
Date Posted: 29 June 2006 at 3:39am
+7 here... 
|
Posted By: caratking
Date Posted: 01 July 2006 at 7:44am
I thought I would at least share some information with you guys, perhaps related or perhaps not... since you have not cracked the case on this issue yet. I had spamfilter running for some time, no issues. Then I started implementing REGEX filter of my own creation. All of a sudden I started having issues much like everyone is describing here. Stuck connections, and in my case 100% CPU usage as well. Obviously the number of incoming connections would start to build up. The problem could only be sorted out by shutting things down and bringing it all back up again. The problem went on for days and days, I was getting quite stressed. Eventually I removed the REGEX filters I had put in place and bang the problem was gone. For fun I put the REGEX back online, and the problem came back right away. At that point I just pulled the REGEX stuff out and have not bothered with them again. The server would process thousands of messages and then start acting up when the regex and some combination of email happened. Something to consider anyway, just try removing the regex filters and see if it goes away or not. This also explains why not everyone has the problem, it may only show itself with some combination of regex filters. |
LogSat wrote:Posted By: jerbo128
Date Posted: 02 July 2006 at 10:56pm
|
Roberto - I have emailed my keyword file in to ya. jerbo128 |
Posted By: jerbo128
Date Posted: 06 October 2006 at 10:29am
|
The stuck connections on the connections tab problem has appeared to be solved, with us anyway. However, the past couple of months, several times we have noticed the opposite problem as before. The connections tab shows nothing, however the "current inbound connections" counter shows that there are incoming connections. I have watched the counter / connections tab for quite a period of time, and it appears that the counter does not ever correct without a restart of the service. Some times these connections appear right after a restart, other times the service has been running for a week or more before they show up. I am running 3.1.3.597 Jerbo128 |
Posted By: mikek
Date Posted: 06 October 2006 at 10:32am
| I am seeing the same thing as jerbo - also running 3.1.3.597 |
Posted By: WebGuyz
Date Posted: 06 October 2006 at 10:44am
|
My stuck connections is still going full bore. We have been having this problem for a few weeks and only a restart fixes it. We finally wrote a script to monitor the log file and alert us when we start getting 'Too Many Connections' we know that the number of connection has hit 40 (our setting in config for max connections) and its time to restart SF because of no free connections If we look in SF connections we see 'stuck' connection about 30-40 connections. Eventually mail stops flowing because there are 0 connections freed. Talk about po'd customers. Probably have to find a way to automatically restart the service since this can happen any time. Was hoping this issue would be resolved by now but I guess not. After restart if we perioically check the connections we see the condition you describing, connections page showing less then the number of connections listed at the bottom of the screen in the status area. We removed all out regex filters per someones suggestion on this forum but it has made no difference. Some type of spam is triggering this but I don't see any way of figuring out what it is. Are other still seeing these issues or is it maybe my environment. Running 3.1.3.597 ------------- http://www.webguyz.net |
Posted By: kspare
Date Posted: 06 October 2006 at 11:23am
| I've started to see this problem too. I am running 3.1.3.590 |
Posted By: LogSat
Date Posted: 06 October 2006 at 12:19pm
|
Everyone, While we've never been able to reproduce the problem, we *may* have found a possible way to avoid it. We've been working a bit in the dark as we could only guess what could be happening. Build 3.1.3.600 (containing the possible fix) was being tested internally to see if there were any unwanted side effects. As there have not been any so far, and as apparently there are several of you experiencing this symptom, we've just released this verison in the registered user area. We'd appreciate any feedback (yeah/nea) to let us know how it works. ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: kspare
Date Posted: 07 October 2006 at 9:39pm
| So far so good. I have 0 stuck connections when I would have had about 10 so far. |
Posted By: WebGuyz
Date Posted: 09 October 2006 at 7:20am
|
I installed .600 this weekend and it looked optimistic Saturday and Sunday but his morning I'm 6 connections short and I'm sure it will climb as the traffic drags on. ------------- http://www.webguyz.net |
Posted By: LogSat
Date Posted: 10 October 2006 at 10:11pm
|
Everyone, We're trying, with ZERO luck, to replicate this. If any of you have at least 10 "stuck" connections, could you please post or email us a screenshot of your "Connections" tab, and the output of the netstat -n command from a DOS prompt? ------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |
Posted By: LogSat
Date Posted: 12 October 2006 at 9:35am
|
Update. We've made another attempt to change the code that counts the connections, and uploaded build 601 in the registered user area. Initial tests in an installation that hade the "stuck connection" issue look promising.
------------- Roberto Franceschetti http://www.logsat.com" rel="nofollow - LogSat Software http://www.logsat.com/sfi-spam-filter.asp" rel="nofollow - Spam Filter ISP |