Read a large TDMS file by MATLAB

[Guosong]

Dear All

Firstly, please let me complain about such stupid format TDMS.

Since in reality, the streaming file is usually large. For me, I recording high frequency data at a sampling frequency 96 kHz with 10 channels. 

How to read part of the data from a large file by MATLAB seems intractable for me now! Since it cost a lot of time to research TDMS file which brings too much inconvenient to the user, it is useless to research a data format.

I appreciate that some one could give help to read a large file by counting from the begining? I just want to do it like read a common binary file using a counter.

If your are a technical support, please do not give the stupid EXAMPLE from NI which only read a small file each time, it crashes with a large file.

In addition, I would like to give one suggestion to NI. Could you please think the user and give a user friendly support?

Thanks a million for the replies.

Best regards!!!!!!

[TCPlomp]

You're pointing in the wrong direction, NI is not making Matlab...

However colleagues have done importing large (> 4 GB) files and it boils down to the same as one would do with any large dataset in any language (LabVIEW, C, VBS, Excel, Matlab), read in chunks, find out what data-set interests you and read the data. It is easier to decode the data by first defragmenting the TDMS file so you have aligned and continuing blocks of data.

I cannot share the code of my co-workers but it took them some time but it works pretty good.

Ton

[YongqingYe]

Hi Guosong,

I understand how you were unhappy when you trying that, I would like to supply some (hopefully) helpful ideas and here I have some questions firstly:

1. You mentioned you read part of the data from a large file, how much of that "part", just several data values, several KB or hundreds of MB? If something like hundreds of MB, I'm afraid it's definitely slower than KB level data.
2. What the problem of reading the part of data, just very very slow or returning any error? If just slow, how long did it take?
3. Have you ever tried any NI software, instead of Matlab, like LabVIEW to read part of that large file, how did it work and if so, which version of LabVIEW you are using?

Thank you.

Yongqing

NI R&D

[Guosong]

1. I would like to extract parts of large streaming files. Normally several MBs. It is better that I could set a counter first and then load it for analysis.

2. Whatever software loads the data, one has to understand this monopolized format? It costs a lot of time just understanding this special format, and do you think that it is worthy to understand a format?

3. I tried DIADem, and I am afraid that it is stupid some time. As a software, it should be considered for the user. Why not set a function that the user can export the data set in ViewMode?

4. The MATLAB example is intractable to load a large file, and why it is designed as such stupid function?

[YongqingYe]

I'm trying to understand you, but I don't have the experiences of loading a TDMS file in Matlab. I don't think the user need to understand everything about the file format if the user just want to simply write and read the file. So I'm curious why you need to understand the file format? If you want to use some basic functionality of TDMS in LabVIEW, it's quite easy to get started, just a couple of nodes you need to use, why did you need to stuggle to understand the file format? But of course, maybe you need to understand some basic concept, like what is a "channel group", what is a "channel", is that what you mean of "file format"?

In LabVIEW, TDMS has a node called "TDMS Read" and it does support you to read from a channel from a specified position and count, like I want to read 50 values from the 100th value. Is that what you want? You of course don't need to read all the data in the file, I guess it should be similar in Matlab, you should be able to specify the index and count when you reading.

[Guosong]

Since MATALB provides toolbox for signal analysis which is usually applied for a large amount of data, it is not applicable with what you recommended as the Labview NODE.

It should be convienent that MATLAB read the file automatically by counting from the begining. 

Why not provide an example which the user could set the starting index and set how many values could be read?

Or why not provide VIs which could log the measurement into common binary file instead of TDMS?