tdms read is slow on the cRIO9024

[tarzanyu]

Hey

I am currently modifying my old program to log data to the TDMS files instead of text files. The file size is much smaller and writing speed is reasonable. The problem is that when I read the TDMS file, it seems very slow (11.293 seconds), and acquires nearly full CPU capacity (97%) for couple of seconds, which will cut the user interface communication. The file size is below 9MB with 37 columns and 30000 rows. I am not sure it is normal or not, just feel little bit hard to believe. The same program working fine on my laptop (1.281 seconds), but obviously it has more powerful CPU (dual 2.5GHz instead of cRIO's 800MHz). Waiting for any luminous idea, thanks.

[Jarrod_S.]

A few ideas and notes:

1. Defragment the file before trying to read it if possible. This makes the data for all channels contiguous, so the read function doesn't have to search through the file to put it all together.
2. Your cRIO probably runs the VxWorks OS, which has a different endianness than Windows. If the file was written on Windows, then data translation might need to be done for the target platform, which will slow things down.
3. The data type input of the TDMS Read function is only used to determine the type of the data to read. The data passed in is ignored. So there is no need to preallocate the buffer here. This won't speed things up, but it's just a note.
4. Are all channels the same length? If not, the read function will have to pad the smaller channels in the 2D array, because all rows in a 2D array must be the same size.
5. Are the channel values written as doubles? If they're I32's or something else, then the read function has to convert them, which will slow things down.
6. If all else fails, you could at least consider reading the file in chunks instead of all at once. If you put a small delay in the loop, the CPU will drop down considerably and background tasks won't be disturbed. This might not speed anything up, but the behavior will be acceptable.

[tarzanyu]

Hey Jarrod

Thanks for the quick reponse.

1,2,3 and 6 are very helpful and will give a go.

Answers for the rest of questions:

4. They are all same length.

5. Yes, they are written as doubles.

I will keep you posted, thanks again.

[tarzanyu]

I tried to use TDMS defragment function and big-endian format file, but none of them seems bring too much difference.

I accidently found out that by choosing decimal mode instead of interleaved can speed the program from 10xxx (ms) to 17xx (ms), which is good enough to me, but it will still kill the front panel communication for couple of seconds. So I guess I will try to read less amount of data each time instead.

[dkfire]

What do you mean by

---

@tarzanyu wrote:

...but it will still kill the front panel communication for couple of seconds.

---

Are you viewing the "Front panel" of the VI runing in the RIO ?

[tarzanyu]

Yes, I am still tuning the program for performance inprovement purpose when it is running on the cRIO, so I am viewing the front panel of the top level VI to check some performance indicators. Everytime when the program opens the TDMS data file, my laptop will lose the connection to the cRIO and retrieve it in about 10 seconds.

[tarzanyu]

I decreased the number of data to read for each channel, and the front panel disconnection problem disappeared. It seems just a cRIO and network hardware limitation issue. Again thanks for all the replys.

[dkfire]

The only limitation is that when you are viewing the Front Panel of a cRIO vi it will have a very low priority in the tasks runing in cRIO, therefor if the tdms read is runing in a highere priority task and using all the CPU, you will see a lost connection in the "Front Panel". 

It is not a preferreded  way to communicate with a cRIO. You should setup your own communication and make a local VI for the Host PC.

Remember that the cRIO also has a limitted RAM, therefor it will not be able to open all tdms files of any size. 

Any data process of tdms files should be done on a Host PC. 

[tarzanyu]

Thanks dkfire, you are definitely right.

I changed my program to use separate FIFO to provide data to the data processing loop, and it works fine and much faster.

I will write a vi on the host PC to do the realtime system performance monitoring using network-published variable or TCP functions.