TDMS Database analogy / general questions

[ijustlovemath]

Recently posted a similar thread, check here for details: https://forums.ni.com/t5/LabVIEW/How-do-I-neatly-replace-my-excel-sheet-logging-with-database/m-p/31...

I'm trying to figure out how I will use TDMS instead of a standard database. I have ~8000 simulations to run, and will likely be running them on multiple computers, storing data all the while. Each simulation essentially creates a 2D data array. Each column of this array is the logged data of a specific variable. In a database, I'd distinguish each simulation as an aptly named record. What's the analogous construct in TDMS?

The way I think it works is that my Data Set is the set of simulations. The properties are just properties of the data set at large. Each simulation will be a Channel Group, with the properties being the specifics of that simulation (the things that make it unique). Finally, the Channels are the variables.

Is this right?

General questions:

1. Is it possible to run separate simulations (channel groups) on diffierent computers and have them combine into the same TDMS file?

2. Can you combine TDMS files?

3. (silly, but) Is TDMS capable of handling ~2GB of raw data?

4. Can you query individual channel groups for statistics on a specific simulation?

5. How about the dataset as a whole?

[Bob_Schor]

When I did a Google search for TDMS, the first hit was an NI White Paper on TDMS that was one week old!  Don't be afraid to use the Web and do some experimentation on your own.

Bob Schor

[ijustlovemath]

That's where I came from. The whitepaper doesn't do a great job of clearing up those later questions/making the analogy to oldschool relational databases.

[Bob_Schor]

I know you love math, but be a scientist -- do some research (search "TDMS Limitations", "Merge TDMS Files") and do some experimentation (build some dummy files and play with Diadem -- you might even be able to find a Diadem tutorial somewhere).  You might not find "the answer to your question" because we don't know (nor care about) your particular dataset.  You can spend way too much time asking opinions about "which is best", when the concept of "best" is ill-defined.

The point is that TDMS might well be something you can use.  Read up about it, try writing some files, try experimenting with the analysis and reporting capabilities of Diadem.  There is probably a good reason that NI spends a fair amount of effort in supporting this format ...

BS

[YongqingYe]

Please allow me to try to answer the questions:

1. Is it possible to run separate simulations (channel groups) on diffierent computers and have them combine into the same TDMS file?

Yes, it's possible. You can  simply read all the binary content of the TDMS files (by using the Binary file VIs) and combine them all together. But please noted that, the group names and channels cannot be the same in different files.

2. Can you combine TDMS files?

Please see the above answer.

3. (silly, but) Is TDMS capable of handling ~2GB of raw data?

Yes, it's no problem.

4. Can you query individual channel groups for statistics on a specific simulation?

Yes, just use TDMS Read to read out the data.

5. How about the dataset as a whole?

Sorry I'm not sure what the exact meaning of this question.

[Hooovahh]

TDMS file combining is fast and simple.  The only consern is as others have said.  Think of it as an Excel workbooks (because it can be)  If you combine two workbooks, one with a workpage named "Data 1" and another with "Data 2" then the combine would create a file with two work sheets, Data 1 and Data 2, pretty simple.  But if you combined two excel files both with a "Data 1" worksheet, but each column is named differently, then you'll end up with one file with one worksheet, with the columns of the second file, after the columns of the first file.  But if you have two files, with the same worksheet name, and the same column names, then the data from the second file will be appended as rows, after the data from the first file.

TDMS is the same way, it combines it in a relatively intuitive manor.  Another consern is for things like properties.  A TDMS property can be written multiple times, and the value of that property will be the last write that took place.  So if you combine two files both that set the "Start Time" property, the value of Start Time in the combined file, will be the value in the second file.  Here is an old example I did on rearranging TDMS channels and groups.  The purpose was different, but it shows how to combine two TDMS files using the command line. Copy /b function.

http://forums.ni.com/t5/LabVIEW/Rearrange-TDMS/td-p/2449740

Be sure and read the second post where I mention why the VI generates a bad file.