Send big data via network communication

[Mathieu33800]

Hello,

I worked on a project on which I had to send data from LabVIEW executable to a Python script. For that, I used TCP protocol because I hadn't lot of data and the communication was each 1 second.

Now, I have to work on another project on which I have to send from LabVIEW to Python a 2D table with 81920 (320*256) values. So I reproduce my VI but what I want is to exchange data arround 30 times per second. With my VI attached, I only am at 16.

I precise that the two codes are on the same machine so a local network is enough I think.

My first question is : do I use correctly the TCP tools in my VI ?

My second question is : is there another way to send big data fast in the same way?

Thank you in advance for your response.

[crossrulz]

You should not be constantly opening and closing your connection.  Make the connection once at the beginning (before your loop) and close it once at the end (after your loop).  This will eliminate a bunch of overhead.  And as long as there is communications happening every 30 minutes or so, Windows will not close your connection automatically on you.

Also, since you know how many iterations your outer loop will be, you should just use a FOR loop.

[wiebe@CARYA]

You'll have to figure out what the bottleneck is fist.

The 2D double to 2D string conversion can be much faster. The Match Pattern is redundant, just change "%-05.4g" to "%,;%-05.4g". The concatenation of the semi-colon can be done there as well: "%,;%-05.4g;". Now that it's only one formatting function, you might consider to use Array To Spreadsheet String. You wouldn't need the 2 for loops anymore. But it's not exactly the same, the rows will be \n terminated. This will be a huge performance boost (could be 100 to 1000 times faster).

Connecting and disconnecting all the time won't help. Can you open once, and then keep the connection open?

[wiebe@CARYA]

Also, you might consider sending binary data. Doubles are 8 bytes, so won't help much. But since you don't have large accuracy needs, singles might do. Now you have 4 bytes per element, without overhead in stead of 9 digits, a point a semi-colon = 11 bytes. That's significant if throughput is the bottleneck.

You really need to test what the bottleneck is. Nothing you do in LV will help if Python is the limit.

[crossrulz · Accepted solution]

You might also have better luck if you just send the data as a flattened string (Flatten To String) instead of formatting the data into a string.  A since Double is 8 bytes.  For each value, you are currently sending at least 12 bytes.  So you can send less data and you will have more accuracy as well.  If you just use the Flatten To String, it will default to having an I32 at the beginning of the data indicating how long each dimension in the array is.

[Mathieu33800]

First, thank you for your answers. If I open and close once the connexion, I don't know why but my Python script only get one message. It is why I put the open/close in the loop. Moreover, the example here have only 15 loops but at the end the program will turn until the Python script is working, so I will send from Python to Labview a message which tell LabVIEW to shut down.

I will investigate the story of flatten strings and array to spreadsheet string that I didn't know before (I am a beginner on LabVIEW).

I come back in the afternoon to tell you if it is working.

I have to think also to the decode on Python of the message receive. The bracket {} was to be sure to find the beginning and the end of the message, and the ";" was to separate the data.

[crossrulz]

I have no experience with Python, but I am pretty sure your server.accept() should be before your loop.

[Mathieu33800]

I have an error if I accept the connexion outside of the while loop. But you're right, it should be outside. I will investigate all this this afternoon. Thanks.

[Mathieu33800]

With the flattering string, iI don't know why but until a table of 100*100 U16 values, it is working normally, but after I don't receive all data. Do you have any idea about what?

[wiebe@CARYA]

You're probably running in to the size of individual TCP\IP fames or a timeout or something. So you need to read data until you encounter the end of it. TCP\IP communication isn't easy.