Slow TCP Communication between PC and RT Desktop

[wmpanderson]

Hello,

So I am having some issues with TCP communication between a Windows 10 Desktop PC running LabVIEW 2016 and a Desktop PC that was turned into an RT Target (also running 2016 LabVIEW software). I am trying to send a flattened 2D Array of U8 integers in near real-time (i.e.: approximately 60 Hz or faster) between the two computers. The premise is to send the array from the RT Desktop to the Windows PC for some additional processing as I have a particular code segment that cannot be run on an RTOS. Then, I have the results of the processing sent back to the RT Desktop for it to analyze and employ in its controlling algorithms. Please note that the Windows PC is NOT the Host computer for the RT Desktop's code deployment, but is instead an additional computer connected into the same network simply meant to provide additional processing ability.

When I first wrote this code a few months ago, I was able to achieve high rates going in both directions. However, after returning to this project a week or so ago, I noticed that reading from the TCP connection back into the RT Desktop had dropped to approximately 30-40 Hz rate. Because I am using this data for real-time tracking and servo-controlling of a machine, this rate is too slow for use.

Details regarding setup and steps already attempted:

RT Desktop PC = Dell Precision T7600 Workstation running LabVIEW's Phar-Lap RTOS

Windows Desktop PC = Dell Optiplex 780 running Windows 10 and LabVIEW 2016

Size of flattened array string = 165507 bytes (this is both ways)

Considered using Network Streams before TCP/IP --> results displayed several seconds delay in data transmission, hence using TCP communication over Network Streams

Tested using different variable types being written into (Shared, Global, Local) --> demonstrated slow down was in the actual Read, not management of the data that was read

Attempted disabling Nagle's Algorithm on the Windows PC --> no noticeable change in Read rate on RT Desktop

Upgraded from LabVIEW 2015 to LabVIEW 2016 --> again, no noticeable improvements

Downsized data being sent back to RT Desktop from 2D U8 Array to Cluster of 3 values (165507 bytes down to 12 bytes) --> caused Front Panel connection with RT Desktop to choke and cut out

Attempted handshake protocol from RT Desktop's Read to Windows PC's Write to limit writing speed --> caused Front Panel connection to choke

Is there any means of speeding up the Read on the RT Desktop that I have not considered? Thank you in advance for any help that can be provided!

[DonhJoe]

Hi,
How did you configure endpoint buffers when you tried with Network Streams? Did you use the Flush function to reduce latency?
Take a look at the "Minimizing Latency" section in the following document: http://www.ni.com/white-paper/12267/en/

Regards!

[Ben]

---

@wmpanderson wrote:

Hello,

...

When I first wrote this code a few months ago, I was able to achieve high rates going in both directions. However, after returning to this project a week or so ago, I noticed that reading from the TCP connection back into the RT Desktop had dropped to approximately 30-40 Hz rate. Because I am using this data for real-time tracking and servo-controlling of a machine, this rate is too slow for use.

...

Is there any means of speeding up the Read on the RT Desktop that I have not considered? Thank you in advance for any help that can be provided!

---

The obvious question is "what changed between when it worked and when it worked OK and when it slowed down?"

If you have to say "nothing" then have you looked at the network usage and the possibility that there is now an additional load on the network. Did you try running it on an isolated LAN with only your machines running?

Ben

[wmpanderson]

Appreciate the response, DonhJoe.

I did look into using the Network Streams' Flush Function like it recommended in that article. However, it seemed that the NS Write couldn't keep up with the needed speeds to begin with. I believe this was due to the size of the data being sent (461x359 pixel binary images). I even attempted to compress these images down to strings, but the Writer side could only achieve execution speeds ranging from 15 to 30 Hz (need minimum of 60 Hz). In addition, when I specifically slowed down the rates to simply test the transmission of the data itself, I noticed that there was a 2-3 second delay between writing it into the network and reading it from the network. I could see the motion between subsequent images on the write-side, and then 2-3 seconds later, I would see the same motion on the read-side. Could this be caused by a setting within the NS Engine? (I wouldn't even know how to change it if it was...).

I also attempted converting the data to different forms to see if it was simply an issue between transmitting an image-type, U8 2D Array-type, or a Flattened String-type, but still I was not having any luck. I had originally thought I was fairly knowledgeable about Network Streams (I'm the one who introduced them to the rest of my team), but I found myself at a loss when it came to communication of this data. Therefore, I figured my next best bet would be to attempt using the TCP Communication protocols would be the next best thing as discussed in this WhitePages document: http://www.ni.com/white-paper/12079/en/ . As a primitive, I figured whatever "fluff" might have been slowing down the Network Streams wouldn't exist in the TCP. Could I simply be missing something obvious?

Regards

[Ben]

---

@wmpanderson wrote:

... Could I simply be missing something obvious?

Regards

---

I believe that stream ride on TCP and if TCP is not keeping up... how can we expect the streams to fix that?

Ben

[wmpanderson]

---

---

I believe that stream ride on TCP and if TCP is not keeping up... how can we expect the streams to fix that?

Ben

---

Ben,

I first attempted using the Network Streams, but when they didn't work, that's when I moved to TCP. I know that the streams use TCP; however, I had hoped that whatever was done to make NS functions more user-friendly was what was causing the slowdown in transmission. Hence my move to using TCP directly.

Additionally, in regards to your other post, I would certainly say that nothing had changed in my setup between when I was forced to set this project aside and when I picked it back up again. At least, nothing changed that I am aware of. No updates were done to the software; no modifications to the code occurred; not even any changes in the hardware involved in this process happened to the best of my knowledge. This would also extend to the network's loading as well.

We use an Asus RT-AC87U router dedicated specifically for this setup so that we can take advantage of its 5 GHz wireless bandwidth to transmit and save raw data onto the Host computer. The Host computer is the only wireless connection, and it uses the NS functions to transmit the raw data from the RT target to the Host for saving. The RT Target and the secondary Windows computer are the only ones directly connected to the router via Ethernet cables. Unfortunately, this setup is essentially isolated to begin with, so I don't think there's any other ways I could isolate it more. I have checked within the router, and no other machines are connecting to it. Could something be consuming the bandwidth in the background instead? How would I even check for this?

When I attempted to send a reduced size data packet (12 bytes) from the Windows comp to the RT, the RT's connection to its code's Front Panel on the Host was choked out. Could this also be an indication that something is consuming the bandwidth in the background?

Regards

[Ben]

If you peg the CPU of the RT system it will fail to update the FP connection being starved of CPU.

If your RT hardware has a video port you could hook up a monitor and see if the CPU is pegging out.

Ben 

[edjones93]

Just a quick query, what can't be achieved on the RT but can on the PC?

What RT is this? (MAX can tell you).

I know my queries are somewhat unrelated but you can run some DLLs on PharLap:

http://digital.ni.com/public.nsf/allkb/0BF52E6FAC0BF9C286256EDB00015230

http://digital.ni.com/public.nsf/allkb/2EA49B05E67DDECF86256F9A006FB315

Additionally each RTOS handles TCP communication slightly differently hence I also ask. What happens if (if possible) you remove the network and connect directly via Link Local?

Best regards,

Ed

[wmpanderson]

---

If your RT hardware has a video port you could hook up a monitor and see if the CPU is pegging out.

Due to some recent bureaucracy, we can only activate certain systems of our machine for the time being. I've been working on this issue using a simulated setup, but I well certainly check into this the moment we can bring the entire system back online again. If the CPU is pegging out, is there any course you would recommend to pinpointing the specific cause behind it?

[Ben]

Profile the CPU usage to find the offending code then analyze same to determine if it can be throttled or requires optimization.

Ben