Please HelP!! Memory Increases with TDMS write

[sayo9394]

Hello guys,
i'm using a TDMS file to store data from 16 accelerometers sampled at 10KS/sec, so you can imagine that the size of the files are going to be huge (greater than 10Gb for 2 hrs or more of recording).
I want to be able to save a subset from that TDMS file, or even the complete file to a different TDMS file. I'm having no problem writing to the file when reading from the accelerometers.

When trying to read from the TDMS file and write to a new one, I'm having a great deal of memory problems, so i'm reading chunks of 5000 samples in a while loop then writing to file.
i have included the code.

attached is the code. Please help as i've been stuck with this memory problems for more than a week now!!

Kind regards,
Simon

[Herbert Engels]

There is a known problem with memory consumption on string reads. Please contact our Product Support Engineers (or your local sales person) in order to obtain a patch release.

In addition to that, with files that large you might want to try storing larger chunks of data, e.g. 1MB or more per chunk. Otherwise, you will accumulate too much indexing information. When you open the file, the indexing information will be read from the *.tdms_index file. If this file is larger than 1-2MB, you should usually use a larger chunk size. That will speed up opening, reading and writing, plus it will reduce memory consumption.

Herbert

[sayo9394]

Thanx Herbert for your quick responce.

I realised i was using the "Build Array" which allocates new memory everytime it's called (i guess!) and replaced it with Replace Subset Array.

I'm currently using labview 8.5 (august release), WinXP SP2

I'm considering closing and reopening the file within the while loop every few Loops, is this a bad idea?

cheers,
Simon

[TCPlomp · Accepted solution]

I don't think opening closing the file will make any difference.
Have you defragmented the original TDMS file?
And I think you can read in chunks of 10 MB with no problem.
Why do you store the time anyway, your waveform will have time information in it.

Ton

[sayo9394]

Ton, mate really i don't if i can thank you enough!!
I stopped logging the time into the TDMS and it solved the problem. you see i'm editing someone else's code, and i've only been using labview for a month now!

Really guys, thank you very much for your help. I will still contact NI to see if a patch exist to storing string or date data to TDMS without memory leaks. (Thanks Herbert) 🙂

cheers,
Simon