How to store data (in tdm files)

[salte]

I'm looking for advice on how to store data in tdm/tdx files.

The main challenge is like this: At regular time intervals there is new data available (say every 10 minutes there are 10 minutes of new data available). All data is time based. So every ten minutes I may create a new tdm/tdx file containing this data segment. However, when i want to analyze the data i dont necessarily want to view 10 minutes of data from all channels, but maybe 20 hours of data from one particular channel.

The way I've achieved this so far is to manually load each 10 minute segment of this particular channel and then add these parts together. This is both time consuming and cumbersome. Is there a better way to do this that I've simply not discovered yet? Is there a better way to store data to simplify this process?

One solution is of course to save data from several 10-minute frames in one file, but seeing as there is a never ending supply of data i cant simply save all data in one giant file, it has to be split up at some point, and the problem will still remain.

One factor to have in mind here is that this is rather large amounts of data (maybe 10GB each day), so the option to simply load all the data into memory goes out the window rather quickly.

Feel free to ask if I've not made myself very clear 🙂

[Brad_Turpin]

Hi salte,

If your test ever ended, and if you had LabVIEW 8.20 or higher, I would recommend using a TDMS file, which handles data appeands flawlessly.  DIAdem 10.1 now also does data reduction and index windowing during file loading for TDM / TDMS / DataPlugin files, so that you could easily load out only the part of the file you wanted to look at.  But since you describe your data acquisition as never stopping and amassing 10 GB per day, I agree that it would be impractical to use only 1 data file.  So we are stuck with some number of files which each contain a part of your measurement.  This approach can have advantages, since you can save operational properties for each "buffer" such as average value, dominant frequency, ambient room temperature, etc., and later on you can use the DataFinder to query out only the data buffers which meet specific conditions based on these properties.  The problem remains to load and assemble data based on multiple files.  This is an old problem in DIAdem, and one for which I have an efficient and I hope satisfactory workaround application.  It does what you describe already doing in the minimum amount of time and with the minimum amount of user interaction, and it can be highly parametrized to suit your particular situation.  The ideal way to start the application would be as part of a ResultsList custom menu.  This would enable you to query out the buffers you wanted, highlight those rows in the ResultsList, right-click and choose your custom menu, and WHAM! the selected buffers for your queried channel(s) are automatically appended together in the DataPortal.  Launching from a ResultsList custom menu would mean that you could skip the file dialog and just read out the data sources directly from the ResultsList selection.

Let me know what you think,
Brad Turpin
DIAdem Product Support Engineer
National Instruments

[salte]

Thanks for your reply Brad. This sounds promising. I'm going away for a week now, but I'll have a look at it when I get back.

The task in question is related to monitoring some operational data. This data is collected continuously (and usually never looked at again), and if something should go wrong we hope DIAdem can be a helpful tool for locating the cause of the problem by quickly being able to generate some short- and long term reports/trends of the values related to the problem (there are thousands of monitored values, hece the amount of data).