LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

WARNING TDMS memory leak in LV 2010

Highlighted

Hopefully this will save someone the headache that I've been through the last couple of days.  I have a very large applicaiton that is running a final verification test on a production line.  In my testing I noticed a memory leak in the application and after 2 days of debug discovered that the TDMS logging is the culprit.  This is very disappointing since I am (I mean was) a huge fan of the TDMS file format and the LabVIEW functions.  The attached VI reproduces the leak by checking and unchecking the Memory Leak checkbox.  Its a bit ugly but I was just coping and pasting the sections from my application and trying to reproduce the issue.  Luckily for me in this particular application I was only writing to the TDMS log files so I was able to eliminate the problem by switching to the gTDMS versions of the write functions.  I found these referenced in another post about TDMS memory leak, but in that case the leak was caused by the indexing and the fact that the SAME file was continuously written too over a very long period.  As you can see in my case, a log file is opened and closed for each "Test".

 

gTDMS link

 

Thanks,

Brian

Brian Gangloff
DataAct Incorporated
Message 1 of 21
(2,505 Views)
Highlighted

Wow I would be quite interested in this issue.  I tested it in 2011 and it appears to be a problem there as well.  Was a CAR made for this?  Was did you talk to any NI representitives in troubleshooting?  I'm just curious about how to avoid this (other than the suggested method of using the alternate VIs).

0 Kudos
Message 2 of 21
(2,493 Views)
Highlighted

This reproduces in LV2010SP1 as well as LV2009.

 

Have you been able to write a new test case from scratch which will repro the problem? I didn't spend much time looking at your code to try and discern what is actually causing the leak.

0 Kudos
Message 3 of 21
(2,491 Views)
Highlighted

@Hooovahh wrote:

Wow I would be quite interested in this issue.  I tested it in 2011 and it appears to be a problem there as well.  Was a CAR made for this?  Was did you talk to any NI representitives in troubleshooting?  I'm just curious about how to avoid this (other than the suggested method of using the alternate VIs).


I found this thread which led me to try the gTDMS functions.  I posted here in hopes that someone from NI will evaluate and file the correct paperwork.  I am interested in how to avoid it as well since many of my other use cases involve writing and reading from the TDMS log file.  It is the writing of the string array data that is causing the leak, but I haven't tested if strings on their own are the problem or writing strings after writing waveforms causes the leak.  I did case out the property writes and the leak was still present.

Brian Gangloff
DataAct Incorporated
0 Kudos
Message 4 of 21
(2,484 Views)
Highlighted

My last reply made me try a quick test and the leak is caused by writing string array data to MULTIPLE channels.  From the first VI "TDMS Memory Leak.vi" the leak can be turned off if only a single channel is written too.  However, if multiple channels are written to with a single write or single channels with multiple writes as in "TDMS Memory Leak (multiple writes).vi" then there is a leak.

Brian Gangloff
DataAct Incorporated
Download All
Message 5 of 21
(2,472 Views)
Highlighted

Hi Brian,

 

I'm one of the developers of TDMS in NI R&D. Well, this is a problem of TDMS which has been complained by some customers. The reason you see the "memory leak" or the memory usage increment is because TDMS needs to bookkeep some information in memory and when you writing more and more data values, the information we keep in memory will keep increasing.

 

There are some workarounds, gTDMS probably is also one of them, but the original purpose of creating gTDMS is to support writing TDMSs on Linux, Mac and other platforms:

  1. Using "NI_MinimumBufferSize" propertie on channels, you can find the details of the help documentation of TDMS Set Property. It can not eliminate this problem, but would reduce the memory usage significantly. Normally we would set it as 1,000 to 10,000.
  2. From LV 2009, if you writing to the file always with the same layout, like same channels same number of data values, you will not have memory increament.
  3. If you are using LV 2010 and later, you can try to play with TDMS Advanced API, this API will not have any memory increasing problem at all.

Thank you!

Yongqing Ye

NI R&D

0 Kudos
Message 6 of 21
(2,449 Views)
Highlighted

@YongqingYe wrote:

Hi Brian,

 

I'm one of the developers of TDMS in NI R&D. Well, this is a problem of TDMS which has been complained by some customers. The reason you see the "memory leak" or the memory usage increment is because TDMS needs to bookkeep some information in memory and when you writing more and more data values, the information we keep in memory will keep increasing.



I'm not sure I understand your explaination.  Every interation of the loop generates a new TDMS file, and closes the previous one.  Why would generating a new TDMS file increase memory usage, if the last one was closed (and flushed in the close)?

 

If I replaced the TDMS writes with a write text file, why would a previously generated, and closed reference, cause my memory to increase on the next iteration?  In my mind the last file was created, written, and closed so all references to it should be gone.  Additionally the increase in memeory is not seen when there is no writing of the string data, after the writing of the waveform data, leading me to believe that there is some issue with writing several string channels, using a single write primative.

Message 7 of 21
(2,427 Views)
Highlighted

@YongqingYe wrote:

Hi Brian,

 

I'm one of the developers of TDMS in NI R&D. Well, this is a problem of TDMS which has been complained by some customers. The reason you see the "memory leak" or the memory usage increment is because TDMS needs to bookkeep some information in memory and when you writing more and more data values, the information we keep in memory will keep increasing.

 

There are some workarounds, gTDMS probably is also one of them, but the original purpose of creating gTDMS is to support writing TDMSs on Linux, Mac and other platforms:

  1. Using "NI_MinimumBufferSize" propertie on channels, you can find the details of the help documentation of TDMS Set Property. It can not eliminate this problem, but would reduce the memory usage significantly. Normally we would set it as 1,000 to 10,000.
  2. From LV 2009, if you writing to the file always with the same layout, like same channels same number of data values, you will not have memory increament.
  3. If you are using LV 2010 and later, you can try to play with TDMS Advanced API, this API will not have any memory increasing problem at all.

Thank you!

Yongqing Ye

NI R&D


Hello Yongqing,

Apparently you did not bother to look at the examples that I provided or read any of the description either.  As Hooovahh has already pointed out, INDEXING is NOT the issue.  The example writes an array to multiple channels ONE time and then the reference is CLOSED.  In the case that does NOT leak, there are multiple waveform arrays written to the file which would require some indexing but the memory does NOT increase.  The problem is when an array of strings is written to multiple channels and the reference is CLOSED.  Unfortunately this type of quick assumption about the problem is why the real issue was overlooked back in 2009.

Thanks,

Brian

Brian Gangloff
DataAct Incorporated
Message 8 of 21
(2,406 Views)

Hi Brian,

 

I apologize that I didn't take a close enough look at you attached VIs and assumed you have the memory leak issue as we always get from customers which happened when keep logging data to the same file. 

 

I have played around your attached VIs and now I have more understanding of your issue, as you explained, every loop you write to a new file and close the reference. We already reported this issue in NI R&D and will do more investigation. I appreciate your post here and help us to find this issue!

 

Please feel free to let us know if there's any other issue/question on TDMS.

 

Thank you!

Yongqing Ye

NI R&D

0 Kudos
Message 9 of 21
(2,394 Views)
Highlighted

@YongqingYe wrote:

... 

Please feel free to let us know if there's any other issue/question on TDMS.

 

Thank you!

Yongqing Ye

NI R&D


 

Hi Yongquing,

 

could you please post a CAR number associated with this bug. The CAR number will allow us to track the bug and let us know when it is safe to start using TDMS again.

 

These may be a nusense to NI but the contractors of the world loose customers over things like this.

 

Ben

 

Retired Senior Automation Systems Architect with Data Science Automation LabVIEW Champion Knight of NI and Prepper LinkedIn Profile YouTube Channel
Message 10 of 21
(2,367 Views)