Storing irregular sample data in TDMS

[xl600]

I have an application where I am capturing waveform data (DAQmx) from multiple analog interface cards at the same time I am capturing irregularly timed data from an Ethernet network.

The irregular data comes in as an array of channel values at approximately 250Hz with each sample set precisely time stamped.  The inter-sample timing is if utmost importance so this data can't be stored as a waveform.  It has to be stored as an array of channel timestamp/value pairs or timestamp/(Array of channel values).

The DAQmx data, of course, is waveform data with only the starting timestamp and delta needed.

So, would there be any suggestions from the community on popular (Recommended ways) to:

1. Define a TDMS file for the irregular data
2. Create a single TDMS file for both the DAQmx waveform data and the irregular data in a common format

Thanks,

XL600

[Hooovahh · Accepted solution]

In these cases I usually have my first channel be "Time" and it is stored as an array of time stamps, where each time corresponds to a value on that index of another channel.  There is a function in LabVIEW on the waveform palette that will take a waveform and generate all the times associated with the Y values.

Oh and so I usually have a new group for each sample rate, so you may have DAQ as a group at the one rate that is regular, and it has a Time column, and another group could be called Serial which has a time and data column in a non regular rate.

[xl600]

That's what I was leaning towards.  Would you know of an efficent way to store an array of channel samples to TDMS as an update to a set of channels within a group?

For example, I have 128 channels coming in every 4ms or so as an array (Just a UDP packet in reality) being pushed into a RT queue.  The array is a fixed size (132 U32):

Time stamp[0]

Time stamp[1]

Time stamp[2]

Time stamp[3]

Channel0

...

Channel 127

I was thinking a simple for loop with multiple TDMS writes (one for each channel after the timestamp was converted to the LabView f128 bit format), but that seems highly inefficent.  I'd like to write in two steps for each sample:

Timestamp

Channel array

Actually, I'd like to empty the RT queue every time (100 arrays deep) and write

Timestamps

Channel Arrays

Thanks!

[xl600]

I answered my own question.  Using the interleaved mode of the tdms write function allows an array of channel samples to be saved.  Since the time stamp is a different data type, that has to be written to a time channel separately.

since I'm saving individual samples on all channels , I've noticed that the tdms file and index file grow at an unreasonable rate.  After investigating, there is a property that can be set on each channel called NI_MinimumBufferSize (labview 2014 allows setting this on groups and files as well).  According to the help files, setting this to a large value is supposed to buffer writes (assuming buffering is enabled on the tdms open call) so that headers aren't created for each write.

The trouble is, it doesn't seem to work.  No matter what I set the property to, it has no effect on the growth rate of the files.  I have to defrag them.

if I manually buffer my writes, it works great and I don't need to defrag.

What's the trick to getting the MinimumBufferSize property to work as claimed in the help files?

I will post a vi example on Monday when I have access to my LabView pc.

[Hooovahh]

---

@xl600 wrote:

What's the trick to getting the MinimumBufferSize property to work as claimed in the help files?

---

That's one mystery I've never understood either.  In theory this should just work but I can never confirm it does.  There are a couple work arounds and you've discovered one.  If you buffer the data your self and then do a write when you have N samples then that one write will help keep fragmentation down.  Another solution is to periodically defrag.  I've had a separete loop running that gets data in a queue to log.  That loop can be configued to close the TDMS file, defrag, and reopen every couple of hours.  Since the data comes in a lossless queue once the defrag is done it goes and handles all the data that was told to be recorded while it was defragging.

Also in 2015 there are some new in memory functions for TDMS.  I haven't tried them yet but the idea is your TDMS file exists in memory, and it is possible that could have some kind of optomizations that make operations faster.  But I think defrag is still one that has to happen on a file, not on a reference.

[xl600]

I found a very clear explanation for the NI_MinimumBufferSize oddity I'm experienceing:

http://forums.ni.com/t5/LabVIEW/buffered-TDMS-file-still-very-large/td-p/1584008/highlight/true/page...

If I transpose the input array myself then write decimated, NI_MinimumBufferSize appears to do exactly what the help files say (Help files even mention this, it just didn't register to me).  It would be nice to see this buffering work in interleaved mode, but transposing myself isn't difficult.

[Hooovahh]

Yeah as long as you aren't in 2011 (or just remember to use always copy)

http://forums.ni.com/t5/LabVIEW/WARNING-TDMS-memory-leak-in-LV-2010/td-p/1874575/page/2

Basically there was bug that if you used transpose 2D array, and then write to TDMS there was a memory leak.

Also thanks for that link, it is probably what I've been experiencing too.

[xl600]

I sure hope NI has fixed this since 2011!

[Hooovahh]

---

@xl600 wrote:

I sure hope NI has fixed this since 2011!

---

They have, in 2012, but not everyone uses the newest version, there are plenty of post with people in 5.x, or 7.x LabVIEW.  7.x is something like 12 years old.

[Surfmase]

Hi Hooovahh, or everyone else,

I know this is old, but I have the same question.  Any chance you could elaborate on this old response?  How would you create an array of timestamps of which each time is not from a waveform?  For example an irregular encoder pulse as a timestamp and corresponding to each frequency measurement? 

Would this be a separate tdms write function from other waveforms with regular sample rates?

Regardless if I write in one or two functions, if I have an array of timestamps, I cannot build an array of another data type with this array; the entire build array function has a data type problem.