TCP/IP large data packets sending problem

[Blue_Perfect]

I may have the same or at least similar problem with both LV8.5 over network, and LV2011 over localhost.

The LV8.5 network problem:

This was logged with wireshark, and the root cause has not been found out.

In this system, we have communication between LV8.5 and 3rd party system over network with small datapackets, but for very long periods of time, and every once in awhile (0.5h-4h), the data packet is registered by LV as sent, but wireshark says it actually has not been sent. Then comes the funny part, the next send command will trigger a burst of data, that seems to me to be the full history of that session in like three packets. Here our 3rd party client will choke, and we commence with a new connection.

Connection is wired, and address is static.

Today I received some logfiles, that look like the same has happened over localhost tcp communication with LV2011 and another 3rd party client.

I will investigate this further, and return with more information.

Sami Karsten

[rolfk]

The description is quite vague but it sounds like the Naggle algorithme, which is standard always enabled for TCP/IP connections. If what you describe is something entirely different I would take some look at the network interface and its driver. The beheviour you see is certainly something below the actual LabVIEW level. The TCP/IP sockets itself are unlikely to expose such behaviour if it is not the Naggle algorithme, which only would leave the packet driver from your network card as culprit. (and yes network card drivers have been known to be buggy, so don't discard that possibility).

[Sam_Sharp]

Yes, it is almost certainly Nagle's algorithm if you are seeing 'bunching' of TCP data packets. We noticed a similar thing with some of our cRIO to PC communications.

Here's some general background information:

http://www.boundary.com/blog/2012/05/know-a-delay-nagles-algorithm-and-you/

Here's some LabVIEW specific information about how to turn it off:

http://digital.ni.com/public.nsf/allkb/7EFCA5D83B59DFDC86256D60007F5839

[Blue_Perfect]

This might be in wrong discussion thread, as we're definitely not sending any large packets, but symptoms seem similar to me.

The network card is registered as Intel 82566DM-2, (driver version 9.12.18.0) and contrary to what I wrote, currently the address is not static.

This shouldn't matter, we have tried both, with same results. We haven't tried another network card. Yet.

In normal operation we can see the data packets arrive in the 3rd party client, and the wireshark-made TCP/IP communication logs show the packets leaving as they should.

This does not look like it was the Nagle's, as the data does not get buffered at all. Rather, it is being sent (and received) directly after the tcp send command.

We have pinpointed the exact moment the problem arises.

At that point, two odd things happen.

1. The data we send does not appear in TCP port

2. Labview sees nothing out of the ordinary, no errors, no nothing. We have no way of knowing (other than wireshark logs) that the data has not actually been sent.

At this point we are going to run into a timeout, as the third party tool is not acknowledging the command we just sent, and we're going to ping it.

At ping, we sent a short string "ping", but the 3rd party client (and wireshark) sees usually 3 large packets of data, that looks to be the full history of that session, including both, the sent AND received data. I can't be sure, if this is truly a full history data, but that's what it seems to be. It's like the data is buffered somewhere, and the buffer is at that time spit to the port. This is easily repeatable, it only takes some time.

--

XP SP3

Intel Core2 Duo CPU

E7400 @ 2800GHz

2,79GHz, 2.99 GB of RAM

Sami Karsten

[robko]

We could solve our connection-problem as a result of a (paket-)burst with a restart of the 3rd-party-program (sender). But I could not find out the circumstances that lead to the problem. I wrote a LabVIEW-mediator to listen to the 3rd-party-port and that got all data pakets. But after sending them to the LabVIEW-TCP-Receiver, same problem occured as described before. I unfortunately could not change the hardware.  

Kind regards,

Rob

[Blue_Perfect]

I return with more information.

LV8.5 problem

Revisiting the problem with more information actually helped me solve it, thank you Rolf and Sam.

Turns out our third party client didn't obey the agreed protocol, and left all of the client sender control data unread.

The control data is for making sure all of the data get's transmitted. This was filling up the server receiver control data sending buffer.

As the buffer was filling, wireshark started noticing zero window ack's and eventually the session buffer (~65kb) was filled, this usually took 1-4 hours.

At this point client reset the receiving connection, and the unread session buffer was released to client.

Removing the control data sending from server receiver (it's obviously not used in the client, and I don't have control over it) solved the problem.

LV2011 problem

Running a simplified communication test did not produce any errors, so the error must lie elsewhere in the code.

Network interface works just fine