converting tdm to lvm/ working with large amount of data

[sam_]

I use a PCI 6251 card for data aquisition, and labview version 8; to log 5 channels at 100 Khz for approximately 4-5 million samples on each channel (the more the better). I use the express VI for reading and writing data which is strored in .tdm format (file size of the .tdx file is around 150 MB). I did not store it in lvm format to reduce the time taken to aquire data.

1. how do i convert this binary file to a .mat file ?

2. In another approach,  I converted the tdm file into lvm format, this works as long as the file size is small (say 50 MB) bigger than that labview memory gets full and will not save the new file. what is an efficient method to write data (say into lvm format) for big size files without causing labview memory to overflow? I tried saving to multiple files, saving one channel at a time, increased the computer's virtual memory (upto 4880 MB) but i still have problems with 'labview memory full' error.

3.  Another problem i noticed with labview is that once it is used to aquire data, it occupies a lot of the computer's memory, even after the VI stops running, is ther a way to refresh the memory and is this mainly due to  bad programming?

any suggestions?

[DFGray]

I assume from your first question that you are attempting to get your data into Matlab.  If that is the case, you have three options:

1. You can treat the tdx file as a binary file and read directly from Matlab.  Each channel is a contiguous block of the data type you stored it in (DBL, I32, etc.), with the channels in the order you stored them.  You probably know how many points are in each channel.  If not, you can get this information from the XML in the tdm file.  This is probably your best option, since you won't have to convert anything.
   
2. Early versions of TDM storage (those shipping with LV7.1 or earlier) automatically read the entire file into memory when you load it.  If you have LV7.1, you can upgrade to a version which allows you to read portions of the file by downloading and installing the demo version of LV8.  This will upgrade the shared USI component.  You can then read a portion of your large data set into memory and stream it back out to LVM.
3. Do option 2, but use NI-HWS (available on your driver CD under the computer based instruments tab) instead of LVM.  HWS is a hierarchical binary format based on HDF5, so Matlab can read the files directly through its HDF5 interface.  You just need to know the file structure.  You can figure that out using HDFView.  If you take this route and have questions, reply to this post and I will try to answer them.  Note that you may wish to use HWS for your future storage, since its performance is much better than TDM and you can read it from Matlab.  HWS/HDF5 also supports compression, and at your data rates, you can probably pull this off while streaming to disk, if you have a reasonably fast computer.
   

Handling large data sets in LabVIEW is an art, like most programming languages.  Check out the tutorial Managing Large Data Sets in LabVIEW for some helpful pointers and code.

LabVIEW does not release memory until a VI exits memory, even if the VI is not running.  This is an optimization to prevent a repeatedly called VI from requesting the same memory every time it is called.  You can reduce this problem considerably by writing empty arrays to all your front panel objects before you exit your top level VI.  Graphs are a particulary common problem.

[medvall]

Hi Sam,

What are you doing in Matlab that you cannot do in LabVIEW?

Best wishes, Marcus

[sam_]

thanks a lot for the feedback

DFGray: I am able store large files now, basically i saved them as tdm, tdx files. later in post processing i reopen the file and change the binary data into arrays, and then load it into matlab. That way i don't have to monkey around with the binary file formats. HWS format sounds promising, i will try using this format from now on.

medvall:  I'm a fairly new user of labview  and i already have all the analysis programs written in matlab.. thats the only reason for the preference..    

[Thor222]

I'm also in the same boat. I have some analysis code already written in C and MATLAB and would like to read in the TDM data directly. As mentioned in this discussion (option 1), I can open up the XML file, get the byte offsets, and read in most of the data just fine. This only thing I can't read are the time stamps in the binary data. The XML file identifies these as an "eTimeUsi" datatype, which seems to be some sort of 128-bit data. Can anyone tell me what the format of this is? I tried doing a few simple things like reading in the data directly as 128 floating point values, but the numbers are junk (assuming they are supposed to be seconds since some date, which I'm assuming to be 1/1/1904).

Thanks for any help!

[DFGray]

The timestamp is a 128-bit fixed point number with the decimal point in the center (64 bit signed integer, 64 bit fraction).  It is (in theory) the number of seconds since midnight, Jan 1, 1904 GMT.  The attached VIs should make the format a bit clearer.  I can't remember whether it is big-endian or little-endian, so you may have to experiment a bit to get it right.

[DFGray]

Try to remember the attachments this time...

[Thor222]

Ah! Perfect. Thank you very much.