Append waveforms in a single TDMS: How to track the time if waveforms are not-continuous?

[dfousek]

Hi !

My application writes 50kHz waveforms, in segments of 1sec, to a TDMS file, depending an externally triggered condition.

Every 1sec-waveform has a t0 with the absolute time. 

Which is the recommended way to store discontinued waveforms, into a single TDMS, without losing the t0 timestamp of each Wfm segment? I tried playing with the wf_time_pref property without success. 

Obviously, I want to avoid writing explicitely the time, given the high rate and amount of storage needed.

Dimitri

[Marco_Ge]

The TDMS save function by itself appends waveforms, so I think that it is not possible to preserve each t0 without any workaround.

You could write each waveform with different t0 as a new group inside the same TDMS. In this way, each session will be separated.

In alternative,you could append data to the same stream, but writing in a separate TDMS channel the infos to reconstruct your t0: the sample index and the relative t0 saved as a double.

Hope this helps.

Marco

[dfousek]

Thanks Marco for your 2 suggestions.

A general comment to them:

The 1st one complicates the reading of the campaign's data, for the obvious reason of having many new groups or new channels (or both).

The 2nd one (adding a channel with just one data - the t0 - to track gaps), is the one I will realize.

I am suprised, however, that the TDMS concept missed this case, which is a common one for data campaigns outside the Lab, in real conditions where devices are stopped and reset all the time, while the collect data for months & years.  

A suggestion for TDMS v3 ??

Regards,

Dimitri

[AuZn]

Just curious: what is the down side of creating new channels or new groups (or even new files) for each seperate measurement? Is there a performance hit in some way, or it simply makes the program a little more complicated?

[dfousek]

Hi AuZn!

Creating new channels & groups is not an option; for a 1year-campaign with hundreds of restarts & resets & (power) failures.

Creating new files, yes, that could be done, however it is not elegant.

My question is: Why am I wrong to assume that this should be a feature of the TDMS structure? After all, are they misleading or not terms like "Absolute' and 'Relative' timestamps, when "absolute" is in fact a "Relative" plus Nsamples x dt?

Bye,

Dimitri

[gangxu]

I have the same problem here. I tried to reconnect the wireless device after disconnected and write the data again with different t0. But the appended data continuously. The ideal outcome is that vi connects all the data chunks with padded zeros (or specified number) for the gap. Hope there's a solution for that.

[Frankay]

It seems like there was no solution on this thread from a few weeks ago. To get more traffic and continue with this discussion I would create a new thread, but it seems like there is no easy way to do this. 

Here is an article that might be similar to your issue as well: http://forums.ni.com/t5/LabVIEW/append-data-to-tdms-file/td-p/1605304/page/2

If you'd like to suggest it on our Idea Exchange you can do that here: http://forums.ni.com/t5/LabVIEW-Idea-Exchange/idb-p/labviewideas

-----------

Aaron F.

National Instruments

[Marco_Ge]

Sorry but I disagree with Dimitri conclusions. 

In my opinion, not closing a TDMS file for one-year-campaign would mean to have probably a huge file to defragment and to manage when opening it or accessing to a single channel. Maybe this is acceptable for very low sampling frequencies.

I think it is more "elegant" to create a new file on a temporal basis (e.g. hourly,daily, monthly...) or on a fixed maximum size basis, then organize a simple data browsing, for example, with a tree control. Additional information on stored data (e.g events or thresholds) could be created on some virtual channels to speed up some fast search without opening each channel data or analyzing all stored data on files.

About groups, I have widely used for large sets of data without any problem.

These were my thoughts.

Regards,

marco

[Kevin_Price]

---

dfousek wrote:

My question is: Why am I wrong to assume that this should be a feature of the TDMS structure? After all, are they misleading or not terms like "Absolute' and 'Relative' timestamps, when "absolute" is in fact a "Relative" plus Nsamples x dt?

---

A straight literal answer: the "S" in TDMS stands for "streaming", i.e., continuous acquisition & storage.  Storing discontinuous chunks of data cuts against the usage scenario that TDMS was designed and optimized to support.

That said, I do understand that the ability to store discontinuous chunks of data can be vital, it'll just require some special care on your part to use a continuous-streaming API effectively for non-continuous storage.  I tend to define my own "time" channel that associates with the discontinuous analog channel.  Each time I'm gonna write a chunk of analog data, I also create and write the corresponding time array, one time element per sample.  There are other more compact options, I just happen to like the simplicity of a separate time channel for post-processing.

I would tend to be against adding features to TDMS to support discontinuous waveforms -- while it might add convenience when used properly, I can also envision a lot of confusion arising from not-quite proper use.

-Kevin P

[dfousek]

Marco, of course I don't want to have a single file for one year...

My application appends 50kHz waveforms, in segments of 1sec, to a TDMS file, depending an externally triggered condition. This high rate is the reason why I chose not to store the absolute time and look for other options.

The TDMS files I'm talking about are hourly files. Another similar application with lower rates produces daily files.

It seems that the simplest (and elegant too..) solution for that is to add a virtual channel with the absolute t0 of each segment. I only feel that this could be an internal feature of the TDMS.

Kind regards