reducing TDMS file size?

[Frenk]

Hello,

I'm streaming analogue data acquired at 32 kS/s to disk in 32768 data point writes using TDMS Write.vi .

A while back I feel that I saw an article about reducing the size of the TDMS files with some tricks. I think one of them might have been not writing unused high significance bits(?).

I can't find that article now and I"m wondering if anyone had suggestions for reducing file size? Right now its about 0.25 MB/s. This is manageable, but it would be nice to reduce this if it were easily possible.

Thanks in advance,

Frenk

[smercurio_fc]

Not sure about the article that you're referring to, but one simple trick is to save unscaled data, with the scaling factors separate, as mentioned towards the end of this thread. This can actually be applied for other data files.

[Matt_W1]

Writing to the file in bigger chunks might help a little  (it'll make a huge difference if your only writing a few samples at a time).

Use the TDMS Defragment vi to get an idea of what kind of improvement you can make (since a defragmented file should be equivalent (or close to) to a file where everything was written in one call).

But if you're already writing in good sized chunks, removing the scaling factor (as has been suggested) will probably be the best improvement (try to use the smallest integer needed to hold the unscaled data).


Matt W

[Frenk]

Hello,

Thank you for the suggestions. Writing unscaled I32 more than cut my file size in half. Much appreciated.

What is the advantage of saving Waveforms instead of unscaled at such a size cost? They must only be useful for short data collections at low rates?

Best,

Frenk

[Matt_W1]

Depending on your hardware (are you using 16 bit or less daqs) you could probably get away with a i16 or u16, which should half it again. And if you want to safe more space you might be able to zip the tdms file after you've created it (defraging the file seems to help with the compressibility if you go this route)

0.25 mb/s doesn't seem like all that much to me (I do some lossless video capture that takes 18.6 MB/s), a 500 gig hard drive (which costs ~$130) can handle 50+ MB/s and can capture at 0.25 MB/s for over three weeks before filling up (that might not be all that good in your case though).

The advantage of waveforms is that they're a bit easier to work with (since you don't have to deal with scaling), and if you need the accuracy of a double (maybe from some form of post processing), there's no size disadvantage to using it.

Matt W

[Frenk]

Hello,

Hmmm... I've run into an unexpected problem. I'd like to scale the unscaled data from DAQmx for some calculations during the data collection, but it isn't scaling the way I would have expected.

My range is +/-10V and the largest unscaled numbers for data filling that range are very roughly about +/-125000. I would have expected them to approach 2^31 (one bit for sign)!!  How can one scale this data?

Also, my saved TDMS file doesn't seem to have the information that "Cont Acq&Graph Voltage - Graph Unscaled Data from File (TDMS).vi" needs to scale it.

Regarding I16, my data is unusual in that it contains a signal of interest that is about 80 dB below the stimuli that are also in the measurement. For that reason I need the full I32 to avoid excessive quantization error in the signal of interest.

Thanks again!

Frenk

[Matt_W1]

+/- 125000 is pretty close to an 18 bit daq (which is -131072 to 131071).

[Frenk]

Thank you for reminding me I'm in the real world!

I'm using an 18 bit 6289 ADC, but my mind was thinking I32......  wonderful!

[Frenk]

Hello,

I just tried again and I'm not quite there...

My input signal was about +/-1.5V and the unscaled values were about +/-95000.

The input range was +/-5 V.

I would have expected the scaling to go like this:

95000 / (2^17-1) * 5 V = 3.6 V   for my signal amplitude (instead of the correct 1.5 V)

Any thoughts? Thanks for all the help,

Frenk

[Matt_W1]

Are you sure it's set to a +-5V range, those numbers work with a +-2V range.

If you look up tdms in the ni example finder, there's examples for writing to and reading from a tdms with unscaled data, which might prove helpful.