Multifunction DAQ

cancel
Showing results for 
Search instead for 
Did you mean: 

Multiple TDMS DAQmx streaming to the same TDMS file

I am wondering why it does not seem to be possible to setup DAQmx TDMS streaming on more than one task simulataneously to the same TDMS file.

This can be done programmatically but it seems to be prohibited when using the Configure TDMS Logging vi, which will result in an error when the DAQmx read is executed.

Message 1 of 7
(5,760 Views)
In a word, performance. The TDMS-DAQ integration feature is designed for performance.

To elaborate, it's important to understand the underlying architecture of the TDMS data format. It is a binary file format wherein a "header" describes the data layout (names of groups/channels, properties, raw offset information). After a header comes data. If a group name changes or the data layout changes in any way in between TDMS Write calls, a new header must be written to disk in addition to the data.

The two big performance considerations when writing to a file from two concurrent operations:
1) The two threads will lock each other out when writing the data (both TDMS and Windows). This means that the two threads wait on each other when writing to disk (which many times is the bottleneck of an application). Ideally, two logging threads should be free running without a dependency between them).
2) A new header has to be written to disk every time the two threads "take turns" in order to describe the new data layout for the accompanying data. That is, if Thread A writes to "Group 1" and then Thread B writes to "Group 2", a new header will be written to disk. These headers can be pretty sizeable sometimes depending on number of channels, channel names, and chunk size (if only 1 sample is written to disk, the header has a much more significant impact than if 10,000 points per header). These headers can hurt your application in two ways: longer time to write since it has to write more data, and the file size can be much bigger than necessary.

For these two main reasons, it is not ideal to write to the same TDMS file from two concurrent loops. I do not recommend that your application do this. Furthermore, I fear that even adding support for this to the DAQmx API would lead unsuspecting customers to fall prey to these disadvantages without realizing the downsides.

If you want everything to be in the same file, you could do post-processing (after the logging is complete, concatenate the files). For TDMS files, this would just take a straight binary concatenation (from command line, for example) where it would copy the data from one file and simply append it to the other.
Thanks,

Andy McRorie
NI R&D
Message 2 of 7
(5,748 Views)

Hi Andy,

 

Thanks for your help in understanding how TDMS is working under the hood.  I suppose I will take your advise on trying to keep the TDMS files seperate.  I did not originally want to do this

because I have so many different tasks as the result of numerous encoder channels which unfortunately each need their own task.  I think there is a huge use case here that NI does not really address very well.  That is were you either would like to aggregate AI and Counter channels together in the DAQmx task (perhaps a new type of composite task would be interesting) or aggregate the data in the TDMS file.  The other thing missing for this type of application is a TDMS like data structure that functions more like a realtime database something like TDMS + SQLlite or Citadel.   Maybe this would be a good task for the SE group to chew on 🙂

0 Kudos
Message 3 of 7
(5,738 Views)

With the composite task idea, I think it's important that within a task, every part of the task has the same timing.  It seems like combining AI and CI would generally produce dissimilar timing.  Additionally, it would get a bit strange when you go to do a read.

 

As to the database setup for TDMS, what exactly would you be looking for?  DataFinder, for example, currently provides querying capability wherein it looks at various properties from TDMS files indexed on your system.  Some of TDMS files' architecture is to alleviate the need for a database (although it's clearly not a complete replacement).  If you can shed light on specific abilities that you're looking for, that might help frame what is lacking.  (As a bit of background, here's a previous post wherein I described some of the ways that TDMS is similar to a database: http://forums.ni.com/t5/LabVIEW/Datastore-options-TDMS-or-Database/m-p/1284680#M533488)

Thanks,

Andy McRorie
NI R&D
0 Kudos
Message 4 of 7
(5,729 Views)

The very common use case is were you need to create a general data logger that can capture, display and log any sort of data a user might be able to configure on a PXI chassis.  That might include AI, DI and counter type data such as encoders.  You can either try to sample at a uniform data rate (some slow channels may be oversampled to acheive this) and then aggregate all of the data to one TDMS file.  Otherwise you will need a TDMS file for each data rate and each task.  A good multirate High Speed TDMS Data Logger Reference design is conspicuously missing.

0 Kudos
Message 5 of 7
(5,716 Views)

Just for clarification, is this use case specifically in reference to your statement:

 

The other thing missing for this type of application is a TDMS like data structure that functions more like a realtime database something like TDMS + SQLlite or Citadel

 
Additionally, please note that what you are wanting to do with multiple tasks/rates to the same file is currently possible and relatively easy - just not with the built-in logging feature.  It does not provide the optimal performance, but if your performance requirements do not dictate that you have to optimize a lot, this might very well not be a problem for you.  Furthermore, you can reduce the number of headers by setting the property NI_MinimumBufferSize on every channel after opening your file.

 

Thanks,

Andy McRorie
NI R&D
Message 6 of 7
(5,712 Views)

<The binary concatination does not seem to work. Using v2 of TDMS. Can you please elorbate on this method of merging TDMS Files.> <-- Remeber to disable prepending array/string sizes to Binary Files. I can't count the number of times this input has messed me up.

 

Thanks,

Brian G. Shea
Certified LabVIEW Architect
0 Kudos
Message 7 of 7
(4,824 Views)