High volume TDMS data analysis

[asukumari]

I acquire data at a high sampling rate (1.5GS/sec) on 2 channels and write them to a tdms file. Each tdms file can reach upto 27.4 GB. I am not sure if this is the right forum to ask, but what computation software is the best one to analyze such huge data files. I tried Python and it takes for ever to load the data to a pandas data frame and do some number crunching.

Any advise?

[AYanez]

Hello asukumari. I agree that Pandas is good for data mining but out of the box it could be high RAM consumption, you need to apply some optimizations to decrease the memory usage and having yet the algorithms available. On the other side, TDMS is a high performance file format, but you should care about fragmentation, the best is to write segments with high amount of data (Big arrays) you can see that when you open a TDMS a tdms index file is created, if the size of this last file is big compared to the tdms file then you have high fragmentation. To analyze TDMS you could have DIAdem or creating a custom software or VIs using LabVIEW. I hope it helps you.

[asukumari]

Thank you!

I do not de-fragment the file during the code execution. Is it wise to include that in my VI or do it manually after have saved the full file on the hard drive.

There is a TDMS_index file associated with every *.tdms file but the index file is of very small size compared to the tdms file. Does this mean there is no need of further de-fragmentation ?

[AYanez]

Yes, if the TDMS index is small then your degree of fragmentation is low so it is not the cause of your issue. What kind of analysis do you need? I am asking this because we need to remember that the algorithm complexity is very relevant when we analyzing huge amount of data, if you can share with us the analysis to do and some small portion of your data we can help you giving you a brief sketch of a VI or direction of the tool to achieve the task. Some common points for optimization often are: Load only channels needed instead of the entire content. In place processing of the arrays. Taking advantage of file properties.