One-point-per-channel mode for TDMS

[malocascio]

Say I have 10 DAQ channels. I grab one point from every channel at the same time. If I want to write this immediately to TDMS (say to prevent loss of buffered data in case of a crash), I grab those 10 points, write them, and flush the TDMS buffer. This results in horrible fragmentation, to the point that some of my files can take 45 minutes to open in diadem if I don't defragment them first (which can also take a long time). It would be nice if there were a TDMS mode that recognized that the sequence will be one point per channel, so that data could be written this way.

It may be a bit of a fringe case, or it may be useful to a lot of people, I can't tell.

[crossrulz]

For your use case, I would use the Stream to TDMS option in DAQmx.  Then you don't have to bother with the writing yourself.

[malocascio]

Unfortunately, that's not an option in this particular case. I'm not acquiring data using DAQmx. It's a complicated system... data is being sent from a cRIO to a PC via TCP/IP messaging. There are several solutions for simpler cases, when data can be buffered or streamed directly as you suggest. Thanks for pointing that out, though.

[StephenB]

You just need to set TDMS minimum buffer size and it will buffer the writes internally before flushing them and prevent fragmentation. http://digital.ni.com/public.nsf/allkb/63DA22F92660C7308625744700781A8D

[malocascio]

Hi Stephen,

I've heard about the minimum buffer size property before, but maybe I misunderstood it (in fact, I've even read that same article). Could you indulge me with a quick example? I don't think I know how to set it correctly for my case.

In my software, I can have anywhere from one to ~30 channels. I acquire one point (a DBL) for every channel, write those points to TDMS, and flush it. If I understand you correctly, I should set the NI_MinimumBufferSize property and then I do NOT use the TDMS Flush vi.

Questions:

• Do I need to set NI_MinimumBufferSize right after opening the file? Or can it be set at any time?
• Do I need to set NI_MinimumBufferSize for every channel, or can I set it as a property of the whole file?
• What value do I actually set the buffer size to? Does it depend on how many channels I have?
• After setting this, will the interleaved/decimated setting matter? In the tests I've done, I was surprised to find that writing decimated seems to result in much less fragmentation.

Thanks for the help. Sorry if this isn't the appropriate forum for this discussion.

[deppSu]

For your questions,

1. NI_MinimumBufferSize can be set any time before your actual logging part (TDMS Write)

2. Currently, you have to set it for every channel, in the next release of LV. Setting it on group and file will be supported.

3. The bigger value you set, less fragmentation will be caused but it consumes more memory during logging.

4. NI_MinimumBufferSize only works for decimated data logging.

By the way, TDMS will do optimization of disk footprint if you keep logging data of same layout (type, count and same sequence of channels). In your case, if you remove the TDMS Flush after each Write, you might see the difference.

[Mads]

In short: The performance can be improved a little if you allow yourself some buffering, but only if you can also afford to lose that amount of buffered data in the event of a power failure for example...In my case I typically have data sampled at 1 Hz. To get decent performance I would need to buffer time periods that are equivalent to hours, days or weeks (the latter would be too memory expensive as well), but I can only accept to lose a few seconds...

Fragmentation in such scenarios is my main reason for not using TDMS. There is a long thread about the details of this here. 

[Darren]

Any idea that has received less than 4 kudos within 4 years after posting will be automatically declined.

[nanocyte]

This improvement would greatly improve single point write performance

https://forums.ni.com/t5/LabVIEW-Idea-Exchange/Don-t-write-the-same-channel-name-to-disk-every-time-...