tdms memory increase during saving

[Risotto]

Dear Herbert,

Thank you for all the information. May be the TDMS is adapted for application with 10000 channels but in our case, we have "only" 26 channels to measure with a low rate of 10Hz but during 1 or 2 days. I understand there is no memory leak with the .tdms, but anyway after 2 hours our RAM is full.

We solve the problem now by creating our own binary format in c#- where we can put EVERY kind of data. The index information are saved in an xml file wich is not bigger than 2kb and the reading and writing is quick (5 sec for 30MB). May be the format will not work for a high number of channels, but we have actually no need for this.

Regards,
Risotto

[Herbert Engels]

Risotto,

I have tried your VI and I'm sure that the memory increase you are seeing is not caused by the indexing information we keep. There must be another problem. I am currenlty investigating what that problem is.

For reference on the size of memory we need for the indexing information, feel free to run some of our examples. While what they are doing is similar to what your VI does, the memory increase is not significant. That is obviously not the case in your application.

One unrelated remark: You can wire waveforms or arrays of doubles directly to TDMS Write. There is not need to convert your data to the dynamic data type.

I will get back to you after investigating this further.

Thanks,
Herbert

[LV_TOM]

I have this "problem" when using TDMS in labview 8.2.1.

I both write to and read from the same file.

I write 5000 samples every 0.5 second (10Khz sample rate, one channel).

When reading this file(part of file, lets say 1000 points) every 200ms, from another loop in the VI, I can see the memory use increase.

I tried to run this VI for a weekend, and that was all.

Monday morning the computer (NI PXI controller actually) ran very slow, but I managed to open perfmon and open the trend showing Labview using 1800MB of the pagefile memory.(total 2000MB)

When I closed Labview, it slowly released memory until it was the same as before starting my VI.

The next thing I did, trying to overcome this problem, was to close the file at a specific filesize, and start with a new empty file.

Shouldn't the memory used for index information be released when closing the TDMS file?

Anyway, it did not help.

Do you have any solution that could free the memory used by TDMS file functions without closing labview?

Tom

[Herbert Engels]

Memory leak
We tested TDMS for memory leaks very thoroughly. Memory should go back to normal as soon as you close the file. If TDMS in LabVIEW 8.2.1 still leaks memory, can you please send me a VI that reproduces this?

Index information
TDMS writes a piece of index information to memory and to disc every time you execute "TDMS Write". The way to reduce this overhead is to store data in larger arrays. If you are doing single-point acquisition and write every single value to disc using "TDMS Write", I can see that you might run into these issues.
Try the following (requires LabVIEW 8.2.1):
For every channel you write, set the property NI_MinimumBufferSize to 1000 or 10000. That will cause LabVIEW to accumulate this number of values in memory before flushing it to disc. Your memory footprint will be much smaller, your files will be much smaller, and reading performance is going to be a lot better. If you read from the same file at the same time, use the same refnum you use for writing. "TDMS Read" will then put out values from disc and from the internal buffer (Note that the buffering works for string values, but reading the buffer at the same time currently doesn't. You would see only the string values that are on disc already).

BTW, the 10000 channels use case is only one of our test cases. TDMS is also used for high-speed applications with few channels. For example, TDMS is used for the SignalExpress logging feature.

Hope that helps,
Herbert