tdms memory increase during saving

[Risotto]

Hi,

I am running Labview8.2 under Windows2000 SP4.

I have an application which is saving data acquired at 1000 Hz from a DAQmx-card to a file. The saving VI is called each 2s from a loop and save the data in a .tdms file. When I follow the acquisition task in the task manager, I can see the memory is slowly increasing. Usually the application as to be ran for several days. One file may reach 500 MB.

It seems to be, that the tdms VI keep somehow information on the data in memory each time it save measurement in the file. I read now some article and issue from the NI Developper and the NI forums. I can understand the tdms-file are optimized for writing and not reading like .tdm. When I whant to open these files, I have to defragment them before and then to wait a long time, because the whole file will be loaded in the memory. But what I don't understand is why the memory is increasing when I save data. The same application was using a C#-dll to save the same kind of data in txt-file before and I never has problems.

Some solutions are to split the data into several files (each hour continue to save data in a new file) but I would llike to know before to do this, if there is the possiblity to remove the ununsed memory from the tdms-VI. With measurement studio for example?

Regards.

Risotto

[Philipp R.]

Dear Risotto

Without knowing you VI it's hard to tell whether this may solve your problem, but there is a 'Request Deallocation' VI which is meant to clear unused memory of SubVIs.
There's an interesting thread on that: http://forums.ni.com/ni/board/message?board.id=170&message.id=142503&requireLogin=False.

Best regards

Philipp Roessler

[Risotto]

Hi Phillip,

I read the link about memory management you sent me. This is really interesting, but I am not sure, this is exactly my problem. I try to explain: in my application I use a tdms file to save my acquired data. The acquisition and the saving of the data are all called from the same while loop. This means I can only use the deallocate memory VI after I am out the while loop. But this is too late and anyway I close the tdms-file after the while loop is finished and then my application stops. No, my problem is that during the acquisition and because of the saving of the data in the tdms, my memory is increasing and after some hours, my RAM is full and Labview crashes.

I sent a example VI so that you can see what appens. Please run the VI and look in the task manager at the labview.exe process. You will see, that the memory will slowly but surely increase.

Regards,
Risotto

[Philipp R.]

Dear Risotto

If you put the 'Wait (ms)' timer VI inside the while-loop (which you probably meant to do in the first place), the growth of RAM used by LabVIEW slows down dramatically; running your program 5 minutes on my machine yielded:
~ timer VI outside while-loop: 187.4 MB RAM consumption,
~ timer VI inside while-loop: 0.5 MB RAM consumption.

Best regards

Philipp Roessler

[Risotto]

Hello Philipp,

Yes you are rigth, I meant to put the timer inside the while loop. I only forget to do it, sorry.

But anyway the memory growthinh stays. If you get 0,5 MB each 5 minutes, than after one day, you get 144 MB. Ok, this is only an example but in the real application, we save bigger amount of data over a longuer time, so that any little increasing of memory is potentially dangerous for the stability of the system.

Actually this example is meant to show, that the problem with the increasing memory is coming from the "TDMS Write" VI. I tried to use "TDMS Flush" to save all the data and I was expecting the internal buffer (used by tdms vis) to be cleared. But I seems no to be like this. Usually a flush command empty the concerned buffer, but not in the case of tdms.

So the question is, is there a way avoid the memory increasing or to empty this internal buffer of the TDMS?

Regards,

Olivier

[Philipp R.]

Dear Olivier

Even if I disable TDMS writing, LabVIEW's memory grows by 0.4 MB.
Have you run a long time (e.g. 24 hours) test to see whether LabVIEW actually crashes?

What kind of computer are you using? I think a decent PC should be able to handle applications which require 200 or 300 MB of RAM.

Best regards

Philipp Roessler

[Risotto]

Dear Philipp,

I suppose the tdms saving was not disabled.

To disable the saving in tdms file, you have to rigth click on the Diagram Disable Structure (showing the tdms VIs) and choose "Disable this subdiagram". Then select the subdiagram without tdms vis, rigth click again ans choose "Enable this subdiagram". Now, you should have the tdms vis in the "disable" subdiagram and no vis in the "enable" subdiagram.

Our PC is a 3 Ghz with 1 GB. Some tests may run one week and the file at the end may reach several gigabytes....

[Philipp R.]

Dear Olivier

It turned out there's a memory leakage in the 'TDMS Write' VI; the same ongoing memory consumption can be seen in the 'TDMS - Write data (time domain).vi' in the LabVIEW examples. So I think for the time being you should stick to the workaround you proposed in your initial posting ...

Best regards

Philipp Roessler

[Herbert Engels]

TDMS files are optimized for writing and reading, even if both occur at the same time. That's why they keep index information in memory while they are writing to disc. Every chunk of data that is written to a channel will add several bytes to that index information. We have run extensive tests (e.g. acquisition of 10000 channels with dynamically changing setups) in order to verify that the index information does not grow to a problematic size.

With that said, the LabVIEW 8.20 implementation of TDMS is not very efficient with single-point acquisitions. Every chunk of data for every channel (and for single-point that is every data point) will add at least 4 bytes of index. In a single-point scenario that might be more than what your actual data is. My recommendation here is to accumulate values to an array of maybe 1000 values each before writing. That will also give you a good reading performance without defragmenting. Please note that LabVIEW 8.2.1 will be able to do this buffering for you automatically.

The TDMS functions do not leak any memory. The memory is released as soon as you close the file.

Hope that helps,
Herbert Engels
National Instruments

[CracKatoA]

UFFFFF!!!!! National Instruments Engineers...We need to Know these details!!! If we don't know that, we can't program!!!!!!!!!IIf we don't know something like that, programing in LabVIEW can turn in putting blocks in a trash!!!!!!