Diagnosing a Memory Problem

[Gregory]

Hi all,

I'm having some trouble with an application that runs for ~30 hours and then pops up an error that says "Could not complete operation because ran out of memory". This pops up automatically from LabVIEW, not through the simple error handler I use to display most errors.

I have tried using the performance profiling tools, and the biggest memory hogs appear to be some 3rd party camera drivers from Thorlabs. They get a large array from the camera, and format that into LV's picture control. 

I'm not really sure where to go from here, does anyone have suggestions for fixing this problem?

Thank you

[RavensFan]

I see two possibilities.

1.  It is constantly growing the array.

2.  And I think this is more probably considering it takes 30 hours, that the drivers repeatedly open a resource to the device, but fails to to close it.  After enough time, it runs out of that resource.

How much can you dig into these drivers?  Are they all pure LabVIEW and are unlocked so you can look at them?  Are they relying on dll's?

[Ben]

---

@RavensFan wrote:

I see two possibilities.

1.  It is constantly growing the array.

2.  And I think this is more probably considering it takes 30 hours, that the drivers repeatedly open a resource to the device, but fails to to close it.  After enough time, it runs out of that resource.

How much can you dig into these drivers?  Are they all pure LabVIEW and are unlocked so you can look at them?  Are they relying on dll's?

---

repeatedly open a resource and closing it will leak memory slowly. I tested that with just repeatedly opening a closing a VISA session as fast as possible over a week-end.

The Picture control is another place where memory can be consumed. Using a shift register to build-up a image is another place to lose memory.

Ben

[Gregory]

Hi Ravensfan,

I did try to watch the memory allocations, and it seems like the image array is resized a lot, but it always gets resized to the same size. The camera software installs a COM object I think, and all the LabVIEW code uses property nodes and invoke nodes to talk to the camera, which I can see. The code that retrieves an image from the camera is attached, if you wanted to see an example of it.

[Gregory]

Hi Ben,

Can you explain how I can clear the picture control to make sure it doesn't build up memory over time? The drivers use "Draw Unflattened Pixmap" with no picture input to display the image.

I have the camera streaming in a separate module, should I just close it and relaunch it every couple hours?

[Ben]

The 2D picture control has a right-click option "erase first" which when set will clear the indicator.

Generally...

You should be using the Profile Memory over a long period of time to try and spot which component is using up the memory. If all else fails, start whacking large portions of the code to ID what part is leaking. 

Ben

[RavensFan]

It looks like it is based on dotNet, so perhaps something is going on inside of that code.

It seems that Memory and Information references get created and closed.  But I don't get a sense that either of those would change from one iteration to another that they would need to be opened and closed.

ImageInfo reference (s32MemID) gets created and closed each time, but it seems like it is possible that would change each time since it is derived from the "GetActive" method.

[Gregory]

Hi Ravensfan,

Thanks for taking a look at the VI. I do have all the camera code in its own separate module, a while loop receiving messages but mostly just grabbing a new image every 100ms. Could I just close and relaunch that module every couple hours?

Hi Ben,

I do have the picture control erasing first. I can try running the VI profiler overnight and see what I get.

[RavensFan]

---

@Gregory wrote:

Hi Ravensfan,

Thanks for taking a look at the VI. I do have all the camera code in its own separate module, a while loop receiving messages but mostly just grabbing a new image every 100ms. Could I just close and relaunch that module every couple hours?

---

I think that is worth trying.  The only question I have and can't answer is whether closing that module is enough to clear out the leaked memory, or is that memory owned by a higher level process such as the LabVIEW environment itself that won't be recovered unless you shutdown all the LabVIEW processes.

My guess is that if this is running all in one process such as the LabVIEW development environment, or a single built .exe, you won't be able to recover memory by just closing out the one part.  But if the separate module was a separate process, such as separate built .exe that you've established communication with, then closing it out and reopening later would recover the memory.

Unfortunately 30 hours is a long time and you'd have to wait more than a day to find out if any change you make actually solved the problem.  Unless you can figure out away to accelerate the memory leak.

[JÞB]

I might suggest fixing the forked Camera Reference

I would expect LabVIEW to do the safe thing and copy the ref rather than reuse it. It won't deallocate the copy until the vi goes idle.   I could be wrong but, my magic 8-Ball has been known to find some humdinger answers