Intermittent TCP delay in RT to RT communication

[Sima]

I have a TCP connection setup between two RT systems, both of which are desktop ETS computers. The VI on RT2 listens for a connection from the VI on RT1, and then the read/write loops kick in. These loops are supposed to run at 100ms. I am sending back and forth "Read" and "Write" arrays, each are 8000 elements in size. These get flattened/unflattened before/after they are sent/received. Most of the time, the loops in both VIs run fine at 100ms but every so often there will be some delay, which will vary from 1 second to sometimes even 12 seconds. Does anyone know why this might be happening?

I forgot to mention that when the loop delay occurs, I don't get any error output from the TCP send/receive or the flatten/unflatten VIs.

Message Edited by Sima on 01-08-2008 04:04 PM

[Amanda_H]

Hi Sima,

Unfortunately, delays in TCP communication can be caused by a number of factors. This KnowledgeBase article details some of the areas to troubleshoot. Also, this thread and this thread include useful information, including this post by BEHaider:

"FYI to those interested - I think I may have solved or at least improved our problem with TCP/IP delays.

Our GUI launches a dynamic vi that handles all TCP/IP functions. The GUI and TCP/IP vi exchange data using datasockets. We had originally set the TCP/IP vis Priority to highest, which, strangely (to us) didn't seem to have any effect. The GUI priority was normal.

What I did recently was to make sure every vi used by the TCP/IP was set to highest priority (some data conversion vis were not), but then I also set the main TCP/IP vi's Preferred Execution system to "data acquisition", and the main GUI vi's system to "user interface". All sub-vi's remain set to "same as caller".

Since making these changes, I have not been able to cause or otherwise observe the problematic TCP/IP delays. I hope this fixes the problem, and maybe this solution helps out someone else with similar "performance" problems."

[Sima]

Thanks Amanda, I will look into those links. I have been doing some testing with different "Advanced Ethernet settings" in MAX for both RTs. I originally had my settings at Autonegotiate and Interrupt, and most of the time my loops would run at 100ms, but would sometimes pause for several seconds.

Does the Nagle algorithm only apply when you are communicating to a Windows machine, or do the LV TCP VIs implement them even among 2 RT OS machines? I have 2 desktop ETS systems, and I am using the supported Ethernet chipsets (Intel 82550).

Also, I came across this link: http://forums.ni.com/ni/board/message?board.id=170&message.id=101490. Do you know whether there are any similar issues in LV 8.5?

Thanks,

Sima

[Sima]

I tried setting the TCP Send Receive.vi to high priority and its execution system to data acquisition as BEHaider did, but I still see the same problem of intermittent pauses of several seconds.

I wasn't able to use the netbench utility (http://sine.ni.com/devzone/cda/epd/p/id/4692) because when I tried opening those VIs, my LV8.5 said they were too old (LV 5.1.2) to convert.

I ran the VIs here (http://digital.ni.com/public.nsf/websearch/7EFCA5D83B59DFDC86256D60007F5839?OpenDocument) where you can turn on/off the Nagle algorithm, but this will only work for communication between PC and RT because you are calling the wsock32.dll. So what happens when you have RT-RT TCP communication? That page says "By default, Windows (and most other operating systems) use the Nagle algorithm by default. ... Under normal circumstances, LabVIEW uses the default (Nagle turned on) when doing TCP communication." Does the RT OS use Nagle, and where would you disable it?

[Amanda_H]

Hi,

Could you post your code so that we can take a look at it? I'm not sure about disabling the Nagle algorithm, but it seems as though it wouldn't work effectively with RT-RT communication.

[Sima]

Hi, here's my code. I've been looking at other posts and talking about optimizing packet sizes, and wondering if that's the way that I should go too. I didn't think that 8000x8(DBL) = 64K would be too much at 100ms, but maybe that is the problem? Ultimately I would like to be sending and receiving at 10ms if that's possible. I haven't found any example code on splitting up the packets and piecing them back together. I've started to write a VI that chops up the string, but putting them back together I think will be a little trickier, since you have to look for the look for a start or header string.

[Amanda_H]

Hi Sima,

Here is a Developer Zone article that measures the TCP output capabilities of a Real-Time target. Unfortunately, determining this for your system is difficult, as it's influenced by many factors, as I mentioned.

[Sima]

OK, I was able to open the netbench VIs with LV 8.2 and then open them up in LV 8.5.

I tested out communication between PC and RT1, as well as RT1 and RT2, and in both cases I saw transfer rate of about 11.5Mbps even up to a maximum payload of 64K (the size of an 8000 array of doubles). I'm not sure if this is going to show me the same effects of what I'm attempting to do.

I am trying sending the 64KB array every 100ms, which means a rate of 0.64Mbps, and if I wanted to send it at 10ms, the rate would be 6.4Mbps. Either one of these rates is still well below the maximum achieved transfer rate that netbench gave me. But what about the intermittent delays that appear?

[Ben]

Hi Sima,

My old brain says I used to be able to turn-off the Nagel algorithm on RT but I can not find a ref to prove it. In fact Robert of NI said in this thread it can't be shut off.

But we are not doomed yet. Have you isolated the two machines such that they are the only two nodes in the LAN?

A I understand the Nagel algor it attempts to optimize throughput by transfering message of optimal size. I do not remember what the optimal size is. So...

I suggest you do some benchmarking of your packet transfer rates using different packet sizes. I BELIEVE that when you get to the optimal size the throughput should peak. Once you find the right size packet, just pad all of your messages to that size. In fact a better approach would be to start with a sting of optimal size and replace those parts that carry your data. At the recieving end you toss the extra data.

No sure if any of the above helps!

Ben

[Sima]

Hi Ben,

I am using a switch and have just my 3 computers connected (Windows PC and 2 RTs). I was curious to see the effect of turning the Nagle algorithm off on the RT to see if that is affecting the communication, because it seems like that's what I'm seeing.

I will test with smaller array sizes. I am still confused as to how I would piece them back together on the receiving end. It seems like you'd have to write your own (higher-level) communications algorithm. I just recently wrote a parser for AK protocol and there you have STX and ETX transmission bytes, so you know what you're looking for. Here, with the data streaming at you in pieces, where is your start packet and where is your end packet? Or is that something you'd have to include on your own?

Thanks,
Sima