TDMS DAQmx decode

[jeff80]

I have some DAQmx measurement data in TDMS format, and would like to analyze this data in Matlab or custom python.  I have read and understand the TDMS File Format Internal Structure documentation, but the documentation claims that "This article does not describe how to decode DAQmx data.  If you need to read a TDMS file with software that implements native support for TDMS (without using any components provided by National Instruments), you will not be able to interpret this data."

I have tried several online solutions: TDMS Reader and ConvertTDMS(v9) at Matlab File Exchange, both of which failed due to interleaved data formatting or presence of RawDAQmx data.  I have also tried (and might still try to debug) a python version (pyTDMS), which might be failing due to poor file format documentation.

I am next going to try the NI solution: the DLL + Matlab.  I haven't gotten into attempting an implementation yet, but from the looks of it, it's going to be a hack.  Particularly because I now have to install multiple Windows SDKs.

I have two questions:

1) Am I missing something?  I realize NI needs to make a profit from their product, but for the documentation to be missing key information and to have to install Microsoft SDKs is inexcusable.  Here in the real world, NI, we have mixed shops and several vendors provide the tools that we need.  Your website claims TDMS works in Matlab, but this is obviously not the case, at least not easily.  And Excel compatibility is great, but not when you have the problems of large data sets that TDMS is supposedly designed for.  NI, I've tended to avoid your products in the past because I dislike graphical programming, but you seem to be specifically preventing interoperability which might give me a new reason.

2) What references (if any) are available regarding this "decoding" of DAQmx data?  What specifically is different if NI code isn't used?

[Aix-La-Chapelle]

Hi jeff80,

I apogolize for the inconvenience caused here.  I believe the method you are talking about in your post is this solution:

http://zone.ni.com/devzone/cda/epd/p/id/5957

I found some additional information about this particular solution in this forum post: 

http://forums.ni.com/t5/LabVIEW/TDMS-file-will-not-properly-import-into-Matlab/td-p/1305678

Unfortunately, I'm not sure what is specifically different if NI code isn't used. Please let me know if any of this helps.  

[YongqingYe]

Hi jeff80,

You are right, the DAQmx data in TDMS file is encoded. When logging the data from DAQ to TDMS, the data stored in TDMS files is DAQmx raw data, and when reading the data out from TDMS files, TDMS is responsible for decoding it and represent to users mainly as double data type. And commonly DAQmx raw data stored in TDMS files always with DAQmx scaling information.

Currently we only support reading DAQmx data from TDMS files by NI softeware products, is that convenient for you to let us know why you need Matlab or custom python to analyze the TDMS files instead of NI softeware products?

Thank you.

Yongqing Ye

NI R&D

[AndrewMc]

I wanted to hopefully explain some of the background information to hopefully answer some of the specific questions that you have.

Please understand that NI is in no way intending to keep a closed system here. TDMS strives to be an open file format with interoperability with several environments. In that mission, NI provides several APIs to read back TDMS files: .Net API, Excel importer, C API, CVI API, and even a Matlab plugin. These provides APIs provide the best user experience as they are designed around TDMS (eg. functions refer to "groups", "channels", "properties", etc).

It sounds like you had an issue with the Excel add-in with large data sets; I knew that this was an issue in Excel 2003 and previous versions as Excel did not support large data sets. I thought with Excel 2007, we update the importer to work well with Excels new capabilities. If not, we should definitely look into that.

The link that the first reply mentions should yield the Matlab API that works with DAQmx scaled data. If you have further problems with it, please make sure to let us know.

I wasn't sure what your concerns surrounding Windows APIs are as I'm not aware of any of our products installing a Windows API. Could you please elaborate if its a particular concern so that we can look into that?

The reason why I mention all of these supported APIs is not to push them because they are NI-provided solutions. I mention them because I believe that they will make your application the most successful.

We document our raw binary layout, but this should only be used if there's not an easier way to get your job done.

All that being said, to answer your second question, the raw data in a DAQmx TDMS file simply consists of data in the same exact format that you get if you were to call DAQmx Read Raw. The output is a binary blob of interleaved channel data from the ADC without calibration or scaling. The process of deinterleaving that data and applying all of the necessary information with the exact numerical precision is fairly complex and error-prone if left to every user. In many ways, thi is why we created the built-in TDMS logging feature into DAQmx - so that users could stream with the best performance possible without having to implement difficult algorithms. DAQmx does not provide documentation for its raw data format.

I do not recommend trying to implement that code yourself if we have a solution that integrates well and does the heavy lifting for you. If you really want to look into some of the details, we can aid you some in this effort.

Sorry for the long post, but I wanted to make sure you understand that there was no ill will from NI in our approach here. I hope your application is successful, and please let us know if you have any follow-up questions.