Allow non-contiguous waveform data to be handled by TDMS

[GregFreeman]

I have noticed a big limitation in TDMS files. When acquiring DAQmx data, often times it is easiest to acquire a 1D array of waveforms for multiple channels, then log this array of waveforms. The problem is that when you have an application that may log, then stop logging, then resume logging again to the same group and channel (such as an application that logs only when an alarm condition is met), the waveforms are automatically appended, rather than showing the gap in time when nothing was being logged. I would prefer that there be some way data logging is managed in scenarios like this, because right now it seems my only options are to 1) convert waveforms to an xy format to be logged, or create a new group every time a logging flag changes to on (which could get quite confusing to a user). I have to assume switching logging on and off is a common scenario, so I'm surprised this hasn't been addressed.

[YongqingYe]

Thank you for the post. And it's right that TDMS doens't show the "gap" when appending waveforms. Each wavefor has t0 (the start time) and dt (the interval time), when appending waveforms to the existing channel, TDMS directly ignores the new waveform's t0 and dt, only keeps the first waveform's. If TDMS needs to show the "gap" then I'm afraid TDMS also needs to know whether the dt is different from the orignial waveform, that  means TDMS needs to bookkeep some information in both memory and the physical file on disk, that probably would affect the performance.

[SeanDonner]

I just ran into this exact issue and came to the very same conclusion that the OP did.  I ended up writing to both a TDMS file to for visualization as well as a CSV file that writes the data and timestamps using the "Get waveform Time Array".    Although you're correct that TMDS will need to do some book keeping, that bookkeeping is extremely minimal during writes.  All it needs to do is see if the current t0 of the input waveform matches the last t0 and if not then create a new (t0,dt) pair in its metadata section.  I can't see this having any significant impact in performance.

[GregFreeman]

---

All it needs to do is see if the current t0 of the input waveform matches the last t0 and if not then create a new (t0,dt) pair in its metadata section.  I can't see this having any significant impact in performance.

---

True, even some of the TDMS VIs in the SVTK query the TDMS file before writing (such as FFTs to see if an FFT channel with the same name exists already because you can't really append FFTs) and it doesn't really seem to affect performance. I think the bigger issue is the plotting/reading of the data. A noncontiguous waveform doesn't really make sense, as a waveform assumes equal spacing between all points in the waveform. So having breaks in time within the waveform is a bit counterintuitive, and i think that will keep something like this from being a built in feature and instead will continue to need to be implemented by the programmer. In my mind I have somewhat determined it is a "would-be-nice-to-have" feature but realistically will not greatly inhance the end product of LabVIEW TDMS and I'm ok with it falling by the wayside. 

[CoastalMaineBird]

I have need of something similar to this.

Whether it's a Waveform data type or not, I don't care, but I need to be able to record a TIME channel alongside several (250+ in my case) other channels.

If the user stops recording Time,A,B,C, and adds a channel D, then the new channel D's first sample SHOULD BE associated with the Time channel that I pass in with it.

IOW, If I have recorded:
T0   A0   B0   C0
T1   A1   B1   C1
T2   A2   B2   C2
T3   A3   B3   C3
T4   A4   B4   C4
T5   A5   B5   C5
T6   A6   B6   C6
T7   A7   B7   C7

and then I change configuration and append a new line:
T8   A8   B8   C8   D8

then I would expect to retrieve D8 along with T8 as a pair meaning that they occurred at the same time.

As it is, instead D8 in effect becomes D0, associated with T0.

[Sam_Sharp]

To be honest, I had already assumed this was possible (we're currently looking at moving to TDMS from CSV files).

I can think of a few situations where this would be useful:

1) Changing the logging rate on the fly (e.g. to capture data at high speed during an event of interest)

2) Start/Pause/Stop logging by the user

3) Device gets disconnected during operation causing a gap in the data

If this isn't possible, it requires a separate timestamp channel which surely isn't as efficient?

[AristosQueue (NI)]

> Although you're correct that TMDS will need to do some book keeping, that

> bookkeeping is extremely minimal during writes. 

But it is also bookkeeping that I believe it would be bad for TDMS to do for exactly the same reasons that I posted here about why it would be a bad idea for a waveform chart to compute non-uniform spacing b.... You would be blending business logic layers with file write layers. I admit that in the TDMS case the problem is less severe than in the indicator case, but the problem still exists.