LV memory leak - How to use windows API SetProcessWorkingSetSize (from Kernel32.dll)

[rolfk]

These are not leaks!!! If they were memory leaks the memory trimming of Windows would do absolutely nothing to them.

It's the memory management of Windows itself that holds on to memory that LabVIEW has asked for and freed since. This is a performance increasing solution but can work really badly when an application allocates and deallocates large amounts of memory repeatedly.

The crashing of an application while calling the mentioned Windows API function is not strange. If the application is idle there is not really a chance that it will crash, but if it is actively allocating and deallocating memory at that moment, the Windows memory management can get utterly confused. That is why this function is under normal circumstances only called on the interactive minimizing of the app.

It's not a LabVIEW thing but can be a problem for any Windows application that allocates and deallocates large chunks of memory repeatedly. I had it in applications as far back as LabVIEW 6 where we had developed an interface to Twain. With each image scanned the memory got reduced by the block of memory allocated. Tests showed that the actual memory allocation causing the problem did not happen in LabVIEW or the Twain interface itself but in the Twain driver that delivered the images. Minimizing solved the memory accumulation, so did calling that function in the end.

This all shows that the problem is not really LabVIEW itself but the way the memory manager in Windows works. Under certain conditions it seems not able to reuse the memory that was allocated earlier and simply keeps going to accumulate new memory to the process. Trimming its size gets the process back to normal memory consumption, but if the memory had been leaked by the app, trimming could not change anything. The trimming only claims memory that is assigned to the application but marked freed by it.

Message Edited by rolfk on 03-21-2010 10:12 PM

[Navne]

Hi Rolf

Thank you for clearing that out - news to me, so yet another lesson learned in the school of computers:-)

All the best

Jacob 

[Enrico_Segre]

---

rolfk wrote:

These are not leaks!!! If they were memory leaks the memory trimming of Windows would do absolutely nothing to them.

Well. Quick test to that.

I took my code, including the thread minimizing it every 10 second, usin cosmin's VIs as described in my previous post, and I downconverted it to LV8.5 ($$#%$#& -- had to do it in two steps via LV8.6, because of another unsolved bug).

Same machine, same hardware, LV8.5.1, 8.6 and 2009 installed on it.

I then run the code, both in the LV interpreter and as compiled executable, both in 2009 and in 8.5.1, with webserver enabled. In the windows task manager, I note down in each case the "Mem usage" and "VM size" allocated to the process (Labview.exe or the executable, depending on the test)

On a remote machine, I open 20 firefox tabs, pointing at the front panels published by my application. The size of each panel is of the order of 800x1000, while as png each image is some 50kB big. The tabs are autorefreshed automatically every 5 seconds.

I observe "Mem usage" and "VM size" on the host of the application, for several minutes. The result is: when my application runs in LV8.5.1 (or was built with), "VM size" stays put around 85MB, while when it runs in LV2009, it grows steadily at a pace of some 16MB/min. During another test, with different graphics on the front panels, I've seen as much as 39MB/min. Such amounts are something in between the cumulative png size and the cumulative pixel count (assuming 1 byte X pixel), but, however you put it and whatever the internal byte count, is what I would call a leak.

I remark, Known Issue #156479, reported first on 8.6.1, which reports the four-letter word. The issue was indeed unknown to me as the previous version of my code used to run (for years) on 8.5.

Enrico 

[rolfk]

Whatever you call it, Windows application trimming has no influence on real memory leaks. A memory leak is a memory block that the application allocated but forgot to deallocate before overwriting the pointer by something else, effectevely loosing any reference to that memory block and consequently not being able to deallocate it anymore. As far as Windows is concerned this memory is still allocated by the application and assigned to it, and the application trimming will NOT touch this memory in any way.

What application trimming does is search the heap of an application for memory blocks that are assigned to the application but not blocked as being used. This means normally memory blocks that the application has properly free'd. As to the reason why this is happening in your case in LabVIEW 2009 and not in 8.6 there are many possibilities but one which has a quite high chance is that LabVIEW 2009 or one of its components was compiled with a newer Visual Studio version, that uses a newer Microsoft C Runtime library that for some (maybe not so) strange reasons happens to be more lazy about returning freed memory to the operating system.

Is it a problem NI should investigate? Yes! Is it a LabVIEW bug? No, not really! If it is a bug at all it is more likely in the Microsoft C runtime than anywhere else but most likely it is not even that but rather a Microsoft C runtime feature, trading lazy memory handling against increased performance.

Message Edited by rolfk on 03-22-2010 03:38 PM

[Enrico_Segre]

Rolf,

I'm afraid that we're not understanding each other here. I'm not complaining about the utility of application trimming. I am complaining about the webserving memory leak, wich is a regression introduced in either 8.6 or 8.6.1, I was not diligent enough to check which. Application trimming is a workaround to it, which has been suggested earlier, which proves ineffective for me.

Enrico

[Navne]

Well - whatever we call it, the fact of the matter is that running an LV application with a published web panel, using the monitor mode, makes for a continuously increase of the memory used by LV until the program suffocates the entire system and crashes..Unless I force "flush" the allocated memory or instruct the user to press the minimize the button every now and then, which is a bit difficult when programming software for unmanned remote measurements:-)

I call that a bug, even though it may not be caused by a "leak"... Using the embedded mode is not really a solution, since it requires the RTE, and i present a solution for mobile phones plus you only have one license, so in order for multiple users to look at the webpage, I need to bleed thru the nose in licensing fees to NI - not an option.. I can open word or excel without increasing my memory usage - I can look at my remote desktop without the system memory getting depleted - however even my LV development environment keeps using more and more memory, until I minimize the last open window.. Is it a LV bug - maybe not, but it is definitely a very poor implementation of whatever memory management scheme used by NI. I'm aware that memory as such is not a concern for the average user anymore, since there are several GB in even a low scale computer, but I consider it to be a very sloppy approach to coding by NI. 

[rolfk]

LabVIEW used to have it's own memory manager back in the Windows 3.1 days as the Windows memory manager back then did not deserve that name at all. They had licensed the memory manager library Great Circle for that. Since Windows 3.1 support was dropped in version 6.0 that memory manager has also been dropped since the Windows 32 bit memory management is usually just fine.

Apparently there are situations where even the Widnwos memory management does not work perfectly and maybe the solution is for NI to reconsider adding their own memory management to LabVIEW. However implemeting or even just licensing such a component into an application has quite some implications, where the licensing costs are the least of the concerns. I still think it could be a Microsoft runtime library problem or something similar, resulting from the use of a specific Microsoft Visual Studio version for the Webservice creation.

Of course NI needs to look into that, but screaming here about memory leak bugs in LabVIEW while it quite obviously can't be that, is not very productive at all.

[Enrico_Segre]

Rolf,

---

rolfk wrote: Of course NI needs to look into that, but screaming here about memory leak bugs in LabVIEW while it quite obviously can't be that, is not very productive at all.

---

Obviously???

My program definitely grows its memory cosumption in a specific instance, termed memory leak by NI, some one year ago, and not yet solved.

NI should also perhaps investigate how come that Jacob's trick is effective for him, but in my case it is certainly not. Further reporting of the bug usually proceeds with providing a stripped down example reproducing it, but since the issue is aready known to NI, I don't think there is the need for me to do it. In my case, I could afford downgrading do 8.5, but identifying the problem and implementing the downgrade has led to a significant waste of time.

Enrico

[rolfk]

I messed up the names. Great Circle was (is?) a memory debugger. For memory management LabVIEW used what I believe was called Smartheap. It was noticebly better than the Windows 3.1 memory manager but did apparently not provide any real advantages to the Win32 memory management and therefore was abandoned in later LabVIEW releases.

Message Edited by rolfk on 03-22-2010 06:01 PM

[Sreedhar_T]

Well I don't know if I am missing out something here. I also tried using the Processhandler.dll as per the Reduce Memory VI to reduce the memory (in my case its a stand alone application).

I made my application run overnight and used the Reduce Memory.vi (to run after ever 3 Hrs).

In the task manager the memory has reduced drastically and its also maintaining at a level which is far lesser than the initial memory.

BUT, when I use perfmon, the memory is not getting reduced at all. It keeps on contributing. Which therefore means that memory reduction is not happening.

I have posted my problem in a different thread PFA the link for your reference:

http://forums.ni.com/ni/board/message?board.id=170&thread.id=484976&view=by_date_ascending&page=1

Im having high hopes to find a solution here. Any pointers will be highly appreciated.