TDMS memory issue on cRIO-9035 (Linux RT)

[Corey.Rotunno]

Hello,

  I have been unable to get around some sort of memory issue I am experiencing during TDMS writes. As I write to the TDMS file, RAM usage on my RT target steadily increases. I thought this was due to some sort of indexing that happens with very large TDMS files so I am currently splitting my files at the 50MB mark. When the file size limit is reached I flush and close the file and start another. Then when I stop recording I flush and close again. However, My RAM usage continues to increase no matter what.

  I have been reading about this issue and I have repeatedly seen references to using the NI_MinimumBufferSize property as a possible solution. I have implemented this but it doesnt seem to be doing the trick either.

  Has anyone been able to overcome this problem or am I going to have to go back to using the build spreadsheet VI's (please no).

  I have included a simple set of VI's that initializes a 16x2500 array to simulate 2500 values for 16 channels and writes that array to file every 50ms. In my actual application the 16x2500 array are reads from my 16ch analog input module reading at 50kHz. The "RT_TDMSMemoryIssue.vi" would be the main VI to launch.

Thanks,

Corey

[Corey.Rotunno]

I should add, the storage location selector is set to "External Drive" by default, which gives a location of /U/. To change it to /C/ change selector to "Internal Drive".

If you are using a windows based RT machine the program will have to be modified to refelect the windows file structure of C:/ blah blah

[Corey.Rotunno]

Looking at it again, my code had a bug in it that would not support logging to /C/. To encourage a response I am posting an image of the main part of my write code. I would love to figure our where all my RAM is going.

Thanks much,

Corey

[YongqingYe]

By looking at the screenshot of the VI, the setting of the property of NI_MinimumBufferSize should be right. 

A couple of quesitons or clues:

1. If you replace TDMS funtions with other writing functions, like spreadsheet, could you see the same memory incresement?
2. If you change the value of NI_MinimumBufferSize, like from 1000 to 100000, the memory incresement is all the same?

[Corey.Rotunno]

1. I have used write to spreadsheet in the past without any problems, however at this recording rate the conversion to strings for the writetospreadsheet VI is too processor intensive, so I have not really tested it in this context.

2. Changing the minimum buffer size does not seem to have much measurable effect on memory usage.

I have simplified my VI to make it easier to follow, and have also switched to the TDMS Advanced tools...still with no luck. In this more simple VI, when the RECORD boolean transitions to true it creates a new TDMS with a time-stamp based file name and writes the first chuck to it. Then it keeps writing the same chunk over and over again. As the RECORD boolean transitions to OFF the TDMS file is closed. At this point I would expect my memory usage to return to something similar to what it was before I hit record, but it does not decrease at all.

 After creating multiple small files (~20-50MB) my memory is really getting quite full. See pics below or the attached VI.

Thanks,

Corey

TRANSITION TO ON (START RECORDING):

NORMAL WRITE:

TRANSITION TO OFF (STOP RECORDING):

[Craig_ · Accepted solution]

This is probably the filesystem buffering the file I/O in memory. Depending on your version of LabVIEW, LinuxRT memory reporting may include these buffers as "used" memory when in reality they are available to applications if requested.

There's a document which discusses this here: http://digital.ni.com/public.nsf/allkb/AC6200D19D23C61586257C8D006E6DC2?OpenDocument

My main question is do you ever see an out of memory error? What happpens if you wait for memory to fill up?

I expect that you'll see behavior where the memory appears to fill up, but everything keeps operating properly!

[Craig_]

Also I notice you are using File I/O in time loops, you should not do this! Always keep file I/O in normal loops.

http://www.ni.com/white-paper/10435/en/

Note the section that says: "Always perform file IO in non-deterministic threads."

[Corey.Rotunno]

Craig,

  You are correct, that is exactly the behavior I see...once I get to 100% it keeps going 98%-100%-98% and so on. It never actually throws an error. It just scares me seeing the reported memory fill up. I am new to the cRIO and RT environment overall. I just got our first cRIO in from NI and I am trying to port a pretty complex and resource intensive application over from the windows/daqmx world so I am being overly sensitive to resource usage. I am trying my hardest to get everything to run on my cRIO-9035....the next step up in processor capability is a huge jump in cost and really would rather not make that jump unless 100% sure I need to.

The article you linked to is an excellent resource, thank you for digging that up for me. I am going to check the memory using the suggested method from the article next, I will let you know how it goes.

Thanks again,

Corey

[Corey.Rotunno]

"Also I notice you are using File I/O in time loops, you should not do this! Always keep file I/O in normal loops.

http://www.ni.com/white-paper/10435/en/"

Craig,

  Regarding the timed vs. regular loop. I will implement this as far as I can. I can move into a normal while loop, however I do not know if sending data over network stream to a "real" computer is going to work for my application. In the end I will be logging 36 channels at 100kHz. Also, the article touches on maybe not recording to file at all if not really needed....this is not going to be an option for me. Our application is used for running and recording data for qualification and R&D purposes, so data files are sort of the whole point.

Thank you though, between these two resources you have sent me I think I am going to be alright. I will mark your post as a solution once I can investigate a little more using the resources you provided.

Corey

[Craig_]

You can definitely log data on the cRIO, it's just bad practice to put those operations in a high-priority loop. 

Good luck!