How can I sample data into small files and work with Matlab?

[Dvido]

Hi,

I am new to LabView. I have a PXI-6133. I need to do a very simple thing:

Sample 8 analog inputs simultaneously at 2MHz for a long time (~30min), save the data and then work with the data in Matlab. 

I am using the "TDMS streaming - Cont Log Data Only.vi" but I have 2 problems:

1. Since I am sampling for a long time, the resulting files are very big. I found several VIs that are supposed to split them but none of them worked.

2. I'm unable to open the TDMS files with Matlab - again, I found several posts on that issue but nothing worked for me.

How can I perform the data acquisition? It can be read to any file as long as it's divided into a few files and can be read by Matlab.

Thanks in advance.

[StephanieA]

Greetings,

You can read TDMS files in Matlab. I'm sure when researching on the forums you came across this page that has examples code on how to read TDMS files in Matlab. If you download the attachment you can run the .m scripts located in the samples folder.

To address the other questions, which VIs were you looking at that break down TDMS files? If you use the VIs do you get any errors? Do you want a certain number of points in each file? Currently the TDMS files are too large to process all of the data at once? You could definitely implement some logic to create new files once a certain number of iterations of your program has run. That would involve using the lower level TDMS file VIs. The DAQmx Configure Logging VI you are using isn't very customizable and you may have better luck using the lower level TDMS VIs.

[AndrewMc]

For splitting data into multiple files, we are hoping to address that application in a future version of DAQmx. In the meanwhile, you can definitely split data into multiples files after an acquisition. The only downside with this approach is that you would end up with bigger files (since the DAQmx data is compressed). I can write you an example that does this when I get a chance.

[AndrewMc]

I wrote an example for you that will split a TDMS file into smaller TDMS files.  Note that the alternative solution here is to read back the data from the single file in smaller sets.  That is, the read functions for TDMS will allow you to read a subset of the file.

[Dvido]

Hi,

Thank you all for your help. I managed to write something myself though I suspect that since I am completely new to LabView it's not very elegant. Is it ok if I post it here and you take a look and tell how is it? It divides the files properly but there is an issue afterwards (next paragraph):

About the TDMS files - I did find the link and downloaded the .dll. The problem is that it works fine with the original (big) file that I sample but after I divide it to smaller files, sometimes they won't open and sometimes they are opened with the data all messed up. I found a workaround by opening them in Excel and saving as .csv but I would like to work directly with Matlab.

[StephanieA]

Go ahead and let us take a look at it. It may be helpful to see what you are doing. Could you also run the example program that Andy wrote. I'd be curious to see if the program he wrote has the same problems when reading in a TDMS file in Matlab.

If you were to read one of the smaller files in LabVIEW using the TDMS file viewer does the data come out as expected there? If so, I'd be curious to see one of the smaller files that is not being read correctly in Matlab.

[bryantt]

hello, McRorie

I have the same problem. My TDMS file is around 3g, and needs changed into around 10M size files. 

I ran your splitFiles.vi , and set the number of samples per file as 5000000, however, I cannot get the results I need. every small file is only 1KB, and no data inside. Can you explain possible problem in this?

Also, I tried to write a vi based on sample vi(Read TDMS File) by adding one "write to measurement file.vi". However, when I set the small file size as 10M Byte inside the "write to measurement file.vi", the first file could be around 20M, and the next few files may be correct as 10M, and then it just stop splitting, edding with a file even much larger than original file. I uploaded my vi here, maybe you can help to find some mistake for me. 

Thanks very much!

Wuwei Mao

[Amartansh13]

Hi Andrew!

Thanks a lot for your code. I want to split my 9Gb tdms file into 9 x 1Gb file. Can I use your vi to split my tdms file without any data loss?

You can also answer to my new post here: http://forums.ni.com/t5/LabVIEW/Handling-large-data-using-tdms-data-logging-and-analysing-it-in/td-p...

Thanks in advance!

[natasftw]

Andrew last visited these forums in 2013.  That's the danger in replying to posts from 7 years ago or bumping threads that haven't had any replies in 3 years.  You're very unlikely to get a response.

You're best off running the code and performing some tests on your own.  It's also unlikely to original author tested your exact use case and you'll want to do these tests on your own regardless.