TCP listen stops listening

[Dennis_Vriezekolk]

Let's discribe the situation first:

I have a TCP/IP server running (port 9503). Several clients (5 to about 40) can connect to this server. The clients use a simple tekst based protocol we developed ourselves. The server starts a dynamic vi for every connection made to handle its requests.

We use a small embedded linux (2.4.18) platform, running on a 300Mhz geode processor. The application runs in the LabVIEW runtime environment version 7.0.

Here's the problem:
Sometimes after a while the server doesn't accept any connections anymore. When I use a TCP/IP protocol analyzer (ethereal) I can see the server isn't listening anymore because the TCP connect (SYN flag) message is replied with a TCP connection refused (flags RST,ACK) message. The server then never accepts any more connections, until the application is restarted, or the pc is rebooted. Connections that have been opened before this problem arises keep alive. Also a different listen (port 9504) is not having the same problems, but isn't used at all by default.

Using a different port didn't seem to matter either. I searched for applications using this port, but it should be free.

We haven't tried LabVIEW 7.1 yet, but I guess this will be the next step. Are there any known improvements in this version?

Any suggestion to solving this problem is greatly appreciated. I will be happy to answer any questions too.

Thanx in advance!

[CoastalMaineBird]

I had a similar situation once - a working program (running on several CPUS solving a common computational problem) would slowly start having TCP errors. Eventually it would stop accepting connections altogether.

Each computer would open a connection to the others in the cluster, obtain a work descriptor, then close it. Then it would do the work, open another connection, report the results, and repeat.

My problem was caused by not closing the connections ON BOTH ENDS. Some resource was being used up by all the opening of connections, and not being freed.

I chased it down (on Win2000) by going to http://www.sysinternals.com/ and getting the TCPVIEW utility.
It shows the current connections and their status.

I don't know what Linux tools would help you.

[Dennis_Vriezekolk]

On the server the dynamic vi that handles all requests from one client closes the network refnum after either a time-out or an error. Also the clients (which run on different pc's) close the refnum in any case before they open a new one. Can 'not closing a refnum' on the client cause problems on the server?

[CoastalMaineBird]

On the server the dynamic vi that handles all requests from one client closes the network refnum after either a time-out or an error.
Presumably, it closes them after a NORMAL termination as well, not just an error.

Can 'not closing a refnum' on the client cause problems on the server?
I don't think so. In my case, each machine was both a client AND a server, and I don't remember which end I had failed to close in my code.

Has the program always shown this problem, or has it recently developed?

Can you run the server on a desktop machine (even with dummy data) and see the same problems?

[Dennis_Vriezekolk]

Yes, the server also closes the connection on normal termination.

We only recently discovered the problem, although it has been running for a while. Also, we discovered it only in one of our largest setups (about 2 or 3 times more traffic than other setups).

Another difficulty is that it isn't consistent at all. It takes a long time to reproduce, and doesn't always give the same results even when circumstances are the same.

We replaced our embedded system by a development system, and that seems to work. The code is now running in development instead of a LabVIEW executable. Also the OS is now Suse (instead of embedded linux) and the pc is way more powerfull than the embedded one.

[Dennis_Vriezekolk]

While checking all open and close situation, I did notice I had the error connected to the TCP close function. Are there situations where the close function doesnt close the connection when an error is supplied to the function?

[CoastalMaineBird]

We replaced our embedded system by a development system, and that seems to work. The code is now running in development instead of a LabVIEW executable. Also the OS is now Suse (instead of embedded linux) and the pc is way more powerfull than the embedded one.

Well, that's a lot of variables in one package - no way to isolate which was (is) the problem.

[CoastalMaineBird]

Are there situations where the close function doesnt close the connection when an error is supplied to the function?

I'm not sure of that - you might want to test it. Are you getting a lot of errors in your program?

In my case, there were a LOT (30k) of connections / disconnections before the problem appeared.
Of course, it's possible that your embedded Linux system has less resources available than my desktop Win2k systems, so it might show up earlier.

It's also possible that the CLOSE CONNECTION function is an exception to the rule (I would design it that way).
Just like CLOSE FILE will close the file whether there's an error IN or not, maybe CLOSE CONNECTION works the same way (I don't know).

[Dennis_Vriezekolk]

I agree that we changed a lot, but we had to make it work, there was no room for any debugging.

[Dennis_Vriezekolk]

I changed the code so the close function doesn't get the error at all. I also tested it on a windows system, and it does seem to close connections when an error is suplied.

I'll make a test case where a lot of connections are openen/closed and see if the problem comes up more often.