TCP write ok -- TCP read timeout

[GregFreeman]

I have an interesting problem. I have two cRIO chassis and a host PC on a network. There is some communication between the cRIOs via tcp/IP(which works fine), but that's just to give you an overview of the system. The issue I'm having is the two cRIOs connect to the host with an architecture very similar to the one in the "multiple connections" example that ships with LabVIEW. I have a listener in one loop which stores connections in a functional global.

Now, the host is getting two valid connections, one for each cRIO, and sends data to each using a for loop to autoindex the refnums which are stored in an array in the functional global. There are no errors when I write to either chassis and one of my chassis reads the data on the other end fine. My other chassis gets a timeout error when trying to read. I am using the same subVI on both chassis to do the read, so it's not as if I may have wired something up wrong.

Can anyone think of a reason I may have a valid connection with both cRIOs, the write would not have an error, but the read on one of them would time out? 

[JackDunaway]

I'll just give you something to chew on, which may or may not be helpful.

I encountered a similar challenge last year, where N remote clients could connect to a host machine. Initially, the host had been written with a "Client Manager" FGV that maintained an array of connection refs (among other things).

I noticed that this system is flawed - a bum client could effectively launch a DOS attack against the other clients connected to the same host, since they all were bottlenecked through the same FOR loop.

Instead, I had the listener spawn a new dedicated server for each client when a connection was requested. Each dedicated server is completely independent, messaging back and forth with the main process manager asynchronously, and it's lifespan is the length of the connection. If the client dropped the connection, the host clone would die, the client would request a new connection, and a new server would be born.

[GregFreeman]

---

@JackDunaway wrote:

I noticed that this system is flawed - a bum client could effectively launch a DOS attack against the other clients connected to the same host, since they all were bottlenecked through the same FOR loop. 

---

Could you explain this in a little more detail? I read some on wikipedia about it, but could a DoS really happen with only 2 connections?

[JackDunaway]

All it takes is two connections! If one - for whatever reason - monopolizes the server, the other client could get starved in a timeout while the server is busy servicing the other client. One client is denying another client service.

Am I being dramatic when I compare your problem to a DoS attack? Perhaps, but the principle remains.

[GregFreeman]

This seems to be exactly what's happening. I close the connection of the non-problem chassis and bingo, the "problem" chassis now reads data. Can you draw up a quick example of how to manage this? What a pain this late in my development that I find out this is the connection issue I have been having.

[nathand]

---

@for(imstuck) wrote:

This seems to be exactly what's happening. I close the connection of the non-problem chassis and bingo, the "problem" chassis now reads data. Can you draw up a quick example of how to manage this? What a pain this late in my development that I find out this is the connection issue I have been having.

---

Is there any way you can post some of your code?

I've used the for loop approach to manage multiple connections without any problems and there was no possibility of a denial of service from one client, because the server only processed one message on each connection before checking the next connection for data.  On the server side there are a couple of tricks to getting this right; the key one is ignoring a timeout error on the read.  As you loop through the connections you want a short timeout on the read, and if there is a timeout error ignore it - it just means that the client didn't send any data, but the connection may still be valid.  I added code to track the last time a message was received on a given connection and waited until that exceeded some limit before deciding to close the connection.  I don't think this describes the problem you're having but maybe it's related, and in any case it's something to be aware of.

[GregFreeman]

---

@nathand wrote:

Is there any way you can post some of your code?

I've used the for loop approach to manage multiple connections without any problems and there was no possibility of a denial of service from one client, because the server only processed one message on each connection before checking the next connection for data.  On the server side there are a couple of tricks to getting this right; the key one is ignoring a timeout error on the read.  As you loop through the connections you want a short timeout on the read, and if there is a timeout error ignore it - it just means that the client didn't send any data, but the connection may still be valid.  I added code to track the last time a message was received on a given connection and waited until that exceeded some limit before deciding to close the connection.  I don't think this describes the problem you're having but maybe it's related, and in any case it's something to be aware of.

---

Unfortunately I can't post it, it is customer's code. I, too, have used this approach before and had no problems, that's why it baffeled me. I think you are misunderstanding a little, the server is writing to two clients when I get the error, not the other way around. The read error is on the client side. Also, I am already ignoring timeout errors on my read. I'll see what happens with the connection when I get hooked up to the hardware today.