05-06-2014 10:30 AM
I'm trying to analyse data which is spread between multiple files over multiple TDMS groups and channels.
The files are contiguous blocks of the same sampled signals (i.e. they are only split to allow efficient storage and visualisation).
I need to analyse each of the signals individually, and so plan to concatenate each channel's data for all of the files, and then run a waveform analysis "en masse".
I generate 288 files per day, each with around 200 channels. Each daily analysis will therefore contain about 8.6 million samples.
However, due to memory restrictions, I need to concatenate each channel individually, rather than using a single continuous file.
I therefore need to use multiple reads to the same files to extract each channel's information.
What is the best way to read from the same files multiple times?
Is it best to Open all files into an array of TDMS references, and then perform a read for each channel, only closing the files at the end of the analysis?
Or would it be better to Open/Read/Close each file for every channel?
Sorry if this is either obvious or has been answered before!
Solved! Go to Solution.
05-06-2014 12:56 PM
05-07-2014 02:25 AM
Hi Cheggers,
Thanks for the quick reply - I had thought that there might be less overhead if I only performed a single open/close on each file, but wasn't sure what memory impact that might have.
The files are 8.6 million samples per channel per day, over about 200 channels. These are acquired continuously in 5 minute time blocks to make it easier to store and manipulate the files later on (I generally only need to review "raw" data in small time blocks, based on the outcome of analysing the whole day's acquisition). Eventually, I suspect the answer will be to run the code in real time to avoid the lengthy post processing, but that is not an option at the moment!
05-07-2014 03:09 AM
So to come back to your question my advise would still be to just open/read/close the files once. I also recommend to check if it is possible for you to change the data type (for example from double to single or even raw data type). This would make the file size much smaller and therefore increase read performance.
cheggers