read tdm fast

[ZIMM]

Hello,

I am trying to read some .tdm data files for a data review tool. My problem is that I find the Read Data Express VI from the storage/data plug in too slow.

I have tried converting these files to TDMS, again using the built in VI, and again very very slow, but the TDMS read function is quite quick in comparison.

Any functions i am missing that I can use to read TDM files faster? Or code snipets?

Thanks for your help!

[crossrulz]

How big is your file?  You could read just parts of the file.

Take at a look at the TDMS Viewer VI (in the File I/O->TDMS palette).

[DFGray]

Under the hood, TDM is a text-based XML file with occasional embedded binary sections.  This gives it cross-platform compatibility and a level of human readability, but makes it inherently slow.  Under the hood, TDMS is a binary format designed to stream data to disk at hardware limited speeds.  If you need speed, converting from TDM to TDMS makes a lot of sense.  However, that conversion will be as slow as the original TDM read.  Other than converting it to another format before using for analysis, there is not much you can do to speed up file access to a TDM file.

[Brad_Turpin]

HI DFGray,

Actually, TDM file format only stores string channels and File/Group/Channel properties in text-based XML.  All other numeric or datetime channel values are stored in one or more binary files, typically one TDX file.  What's more, TDM files usually have all the binary values of a given channel in a contiguous block, which is optimal for reading speed.  The TDM header file is typically very small and contains XML fields that point the reading software quickly to the start positions of each data channel within the binary TDX file.  The TDM header file only slows reading when it contains very large string channels such as uninterpreted CAN messages stored as hex values in ASCII representation.

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[Brad_Turpin]

Hey ZIMM,

Would you be willing to post one of these TDM files for us to take a look at?  It may be something inside the file that is causing the slow load speed, or it might be something in the particular coding approach you're using, or it could be a fundamental limitation of the file format.  I'd like to find out.  If you're not comfortable posting your data to the Discussion Forum, you could email it to me directly at brad.turpin@ni.com.

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[Brad_Turpin]

Hi ZIMM,

One more question, what software program writes these TDM files?  LabVIEW, DIAdem, CVI, or some 3rd party software?

Brad Turpin

DIAdem Product Support Engineer
National Instruments

[ZIMM]

my apologies for the slow response, I was out of town for a bit.

For clarification I am loading in both the TDM file and the corresponding TDX file. TDMs are ~2-4 MB and TDX are typically around 250 MB but can be larger in some cases, up to a gig, though I havent played with any of those.

I can not share any of these data files unfortunately. All of the data is colleted and written with labview as singles, except for the header info such as channels names and such.

I am trying to read this data back into labview. I am currently using the data storage express VIs.

I could certainly read smaller chunks of data at a time, but at some point the whole set will need to be looked at so not sure if that actually buys me anything in the end.

[Whit-E_Wall-E]

In LabVIEW the recommended way to read the TDM files is using the Express VI.  You can also find an example in our NI Example Finder.  You can get there in LabVIEW by going to Help >> Find Examples >> Browse Tab >> expand out Fundamentals >> File Input and Output and then look at the "Read TDM File.vi."

Also how slow is it taking to read the file?  Does it generally take about the same amount of time for the same size file or does it very each time the vi is run?

[ZIMM]

generally the same amount of time for similar sized files, but that time is on the order of minutes (for the size of file i give above) whether i first convert to tdms or read as a tdm.

[Whit-E_Wall-E]

What are your computer specifications?  (Processor, RAM, OS, etc.)

Have you tried to read the TDM files on another computer, do you see the same behavior?