Dealing with large amounts of data

[Sima]

I was hoping to get some input from the DIAdem experts on a topic posted in the LV forum:

How do you handle and distribute huge amounts of data?

Thanks!

[Brad_Turpin]

Hi Sima,

Wow, that was an extensive thread.  It seemed to me like the suggestions eventually settled down to what I would have said myself.  I definitely recommend streaming to a TDMS file.  The TDM Excel Add-in works with TDM and TDMS files and works with all (modern) versions of Excel, including the newly expanded Excel 2007, and it is a free download.  Furthermore, there is an API for reading and writing TDM(S) files from C code, which can be used in other programs, so the data format is pretty open.  If the customer has the money to invest in a tool, then DIAdem would be an ideal fit for processing these large data sets.  DIAdem can easily load specific time windows from the large data set and/or load the data in reduced format, say 1 out of every 100 data points, or the average of every 100 data points or the max and min of each set of 100 data points, etc.  DIAdem can also easily load only 5 or 6 channels out of the total 300, so there are lots of ways to make the data consumption easier, and DIAdem handles large data set very well even if you simply drag them in and graph them.

I actually liked the idea that one of them suggested of building an executable out of the TDMS File Viewer VI on the TDMS palette.  If your data consumers are willing to download the LabVIEW Run-Time Engine and run such an executable, they would be able to look at different time windows of data in tabular and/or graphical format.  Check out the <Settings> button at the bottom of the VI.  This strikes me as a very good basic data viewer solution for no money down.  Note also that with TDMS you would be able to easily capture any descriptive information as properties (Operator, Gain, Humidity, Description, ...), and with the TDMS File Viewer you are also able to look at all of these properties at each of the 3 hierarchy levels (File, Group, Channel).

Brad Turpin
DIAdem Product Support Engineer
National Instruments

[Sima]

Thanks Brad. I have some more questions.

Can you use the TDMS file VIs in combination with other file VIs, in particular, the Get/Set File Position and so on? I've currently got a circular file buffer that writes to a simple binary file, and have been contemplating converting over to TDMS. I'm just concerned I can't access the file byte positions with a TDMS file in the same way as a binary file.

My next question: we provide our customers with a few options in report generation (plain text, Excel, and in the near future, DIAdem). We currently log the data in plain text and then perform the post-processing to pretty up the reports. I don't think that's the best option since ASCII files tend to be much larger. Now that we'd like to add DIAdem to the mix, we're rethinking our data logging file type. What do you suggest would be the best type of file to start with if we'd then like to have the option to convert to text, Excel, or import to DIAdem? I'm going between a simple raw binary file, or TDMS. I have a feeling as to what you might say!

[Herbert Engels]

You cannot set a file pointer in TDMS. You can do circular buffering with a simple trick though: by using two files. You write to the first file until it contains the number of values you want to buffer. Then, you close it and write a second file. Once the second file has the same size, you replace the first file and start over. When reading from the buffer, of course you will need data from both files most of the time, so you need some code for that. On the other hand, if you use a file format that allows you to reposition the file pointer, you will have to get data from different positions of the same file, so the overhead of both solutions is pretty much the same. The only drawback of the TDMS solution is that it requires twice the disc space. But disc space is cheap ...

Hope that helps,
Herbert

[Sima]

That's too bad about the file pointers. It would've made for a much nicer transition into TDMS files. How about just post-processing the binary file and converting into TDMS? I know I know longer benefit from the streaming, but perhaps I can at least get the indexing/DIAdem benefits??

[AEDavid]

Hi Sima,

Two ways to convert this binary file into TDMS would be:

• Use a dataplugin
• Use the DIAdem connectivity VIs (specifically, the DIAdem Write Header File.vi).  This is also available in a C DLL.

Introduction to the LabWindows/CVI TDM Streaming Library

Reading and Writing TDM/TDMS Files

[new_programmer]

Hello Brad,

Attached, I have a VI that uses NI-Scope (NI-5122) to digitize and store data.  I have added some TDMS functions in order to save waveform data in binary format ( our acquisition time is around 3 hours and the files become very large)  As you can see, I am using TDM file viewer to view the data, the problem I am having is that I can only see the voltages and not time stamps.  I am interested in viewing and analyzing timestamped data.  I do see that the graph contains both voltages and timestamps and idealy I would like to see both in the value tab.

Please let me know what is the best way to see both sets of data. 

Thanks!

[Brad_Turpin]

Hi new_progr…,

Your LabVIEW code looks good to me, adding buffers of waveform data with the TDMS Write.vi is standard procedure, and this does indeed capture both the vertical measured values and the horizontal timestamp values and save them to the TDMS file.  The only problem you're having in the TDMS File Viewer is that it does not show you the timestamp information in the table of values.  If you are willing to build your own custom file reader, you could easily add a "Get Time Array.vi" to read out all the time values and display them in your own table next to the vertical measured value.  If, on the other hand, you wish to continue using the TDMS File Viewer, which I can totally understand, then I think the only way of achieving your stated goal would be to use the "Get Time Array.vi" inside the acquisition loop and save two channels of data to the TDMS file.  This is redundant, of course, since you already have the time information stored as t0 and dT properties in your measured TDMS channel, but it would give you the display in the TDMS File Viewer that you want.

Brad Turpin
DIAdem Product Support Engineer
National Instuments