Help: Diagnosing a Memory Leak / Bottleneck

[TheRealNeil]

So I have a project in which we've recently discovered a memory leak / bottleneck. The observed effect is that the memory footprint on Windows Task Manager / Resource Monitor will continue to grow over time. Currently we are unsure if the cause is because of a bottleneck and not being able to process data fast enough, or a memory leak to do some unreleased memory.

I'm unable to post my source code here.

This application generally takes up 30-40% CPU, so that is to say we don't believe the CPU to be the source of the bottleneck.

I'll detail some measurements / information we've gathered / things we've tried to uncover this memory leak.

1. Running Tools > Profile > Performance and Memory
   1. This tool gives Avg / Min / Max bytes and also Avg / Min / Max blocks. None of these we observed to increase over time
2. Running Desktop Execution Trace Toolkit
   
   1. This gave us a good list of *what* was happening, but gave no insight into the increasing memory footprint
3. Running Tools > Profile > Show Buffer Allocation / Profile Buffer Allocations
   1. Neither gave any insight into the increasing memory footprint
4. Running LabVIEW Task Manager
   1. Gave no insight into memory footprint increase
5. Take measurements of timing of system
   1. We gathered a lot of information for both internal processing / display to the UI, and also for logging. We measured how frequently we get data in, and how long it takes to process and log from start to beginning. From our measurements, we determined that data is able to make it through the pipeline fast enough without bottlenecking.
6. Search for unclosed references (found none)

My question becomes - what can we look at next to help uncover the source of this issue? Is there a better tool available that I haven't tried yet? Is there a way to search the project for likely culprits?

Thanks so much in advance!

[santo_13]

A reference leak can be identified using Desktop Execution Trace Toolkit (DETT). Since you already did that and there was no reference leak.

Inefficient implementation of array operations may lead to the observation you described, to improve memory footprint, it requires significant experience and patience to track it down and fix it.

An inefficient implementation can still be faster by consuming a lot of memory, an efficient implementation may not be fast but limits memory footprint. The key takeaway is you compromise one or the other, you may not have the best of both worlds without significant effort.

Now, you've not been telling what is the application and what it does, even with that info, we can only guess and not fix it. You might have to share the whole source code for the folks to dig through for inefficient memory operations.

If your source code is large or cannot be run offline or confidential, it just narrows down to hiring a LabVIEW consultant to optimize your application.

[Michael_Munroe]

<<This application generally takes up 30-40% CPU, so that is to say we don't believe the CPU to be the source of the bottleneck.>>

This statement ignores the fact that all modern processors have 4 or more cores.  Unless your code is always in a parallelized loop, 25% is running full throttle.  This still may be a bottleneck.

Check the profiler for slow VIs to see if any are taking longer than the required loop time. 

if none of the subvis are increasing in memory, the culprit is most likely the the top-level VI.  First search for all queue operations and check for issues. Add the VI to check the size of the queue to troubleshoot.  Next search for array operations like build array, initialize array and index array.  You may be surprised at what you find.

if you are managing large arrays, try using the inplace element structure instead.

[TheRealNeil]

Thank you both for your input! You've given me a few more stones to turn over.

[JÞB]

It sounds like you are running the right set of tools to locate any poor memory management.  That leaves bottlenecks. 

The most common.source is an overworked UI Thread.  With larger projects you absolutely must limit the visibility of "Everything"  us subpanels to display details when needed rather than all the time. USE pop-ups for configuration settings.  GET RID OF TABS. Limit redrawing of splitter movements and FP resize events.  Use Defer FP Updates method to batch process transfer buffer to FP object data.  Fix overlapping FP objects ( find them with VIA) Captions and labels increase object footprint even when the labels and Captions are hidden. We often forget that a redraw requires any overlapping object to redraw too really bogging the UI Thread. 

Watch for CLFNs to threadsafe code an allow to run in any Thread.  Avoid Document Object Model filetypes w3c hasn't published a threadsafe version of DOMUserdefref.dll yet limit updating values by property node.  Limit updating values at faster than eyeball speed or less 5x/sec is really fast.

Essentially,  Your user just can't make any sense of buckets of data being dumped into their sight.  That's why they are using a computer to make sense of it.

[Defaphe]

Hi,

Last time I got such an issue I've tried to put some part of the code into a disable structure starting from the global loop and at the end a tiny sub vi.

In this process I could see the moment a memory leak or CPU usage has risen and tracked down the culpit.

If it can help you.

[TheRealNeil · Accepted solution]

Hi all, just wanted to post an update.

I've gone through many of the above suggestions and it definitely helped improve the memory leak, so thank you for all of those ideas!

We still have a memory leak in our system and my guess is that there many small memory leaks happening throughout the system. This code base is quite large, not well documented and not written by myself so spending the time to uncover them all is an activity for another day.

Thanks again all!