NI Linux Real-Time Discussions

cancel
Showing results for 
Search instead for 
Did you mean: 

Memory Leak in File Writes?

It's pretty vague, but maybe vague is useful here as it gives us wiggle room to provide a best guess at the most useful stat. I'll ask around here.

0 Kudos
Message 21 of 27
(4,415 Views)

From our perspective we don't really care what the underlining OS is doing, its a LabVIEW API that runs accross multiple targets PXI, cRIO, cDAQ on multiple OS's. We would expect it to function similarly accross all targets. We use this property on VXWorks cRIOs and RT PXIs with no issue, it shows if we have any kind of memory leak in our software. But if we use this on RT Linux any data that is written to file reduces the available memory reported, rendering it useless.

The crucial thing for us is for this property to work the same accross all targets. From just the property name I would expect it to report the amout of unallocated RAM in the system so very simplisticly: Free Physical Memory = (Total RAM - (RAM used by OS + LabVIEW Application RAM)). And this is how it seems to work on vxworks and pharlap targets. This way it should show any memory leaks in the LabVIEW application as well as any that may occur at the OS level.

Thanks for the quick responses, it's nice to know some NI people are looking into it.

Jon

Jon Bowers
LabVIEW Architect, TestStand Architect
Message 22 of 27
(4,415 Views)

From just the property name I would expect it to report the amout of unallocated RAM in the system so very simplisticly: Free Physical Memory = (Total RAM - (RAM used by OS + LabVIEW Application RAM))

Right, and that is in fact what it reports. The problem is that "RAM used by the OS" goes up and down more or less independently of what your applicaiton is doing, because the OS's memory management is significantly more complicated than on Phar Lap and VxWorks, as you probably saw in the links in my earlier post. In other words...since a lot of people expect it to report the calculation that you described, we made it do that, but as you noticed, it's not a very useful value on Linux.

My understanding from talking to customers is that the most common use case of these metrics is memory leak checking. On Linux, for that purpose, the thing you really want is something like "memory used by the LabVIEW process". Repurposing the "Free Physical Memory" metric to report the "memory used by LV" statistic instead would be a pretty drastic change from Phar Lap and VxWorks -- too drastic to just handle with a rename. So we'd probably have to add a new metric. And, because LabVIEW is not a process on Phar Lap and VxWorks (it is a collection of threads that runs in the kernel; for this purpose, it is part of the OS), the "memory used by LV" stat would not be practical to report on Phar Lap and VxWorks. It would have to be "memory used" (period) or something like that. Maybe we could just call it that and explain the distinction (system-wide on Phar Lap/VxWorks vs. process-only on Linux/Windows) in documentation. Thoughts?

0 Kudos
Message 23 of 27
(4,415 Views)

I would expect the amount of RAM the OS uses to go up and down (but assume it would be broadly flat over time). I think ther is a more fundamental error in the way the memory is reported using this property. It seems to have been fixed in the distributed system manager, as that reports stable memory usage when writing to files.

FreePhysicalMemory.PNG

The attatched image is what I get when writting files and reading the FreePhysicalMemory property. Where the graph is flat is where I have disabled file writting. It then bottoms out at around 3MB. Memory is only recovered when the files are deleted. DSM reports available memory usage as flat while this VI is running.

DSM.PNG

Here is the code I used for this test.

FreePhysicalMemory VI.PNG

I would be happy if it just reported the same as the DSM.

Changing it to report the LV process memory usage would mean it is difficult to know when the program will run out of memory as you don't know how much the OS is using (we often reboot the application if available memory drops beloow a certain value).

Regards

Jon

Jon Bowers
LabVIEW Architect, TestStand Architect
0 Kudos
Message 24 of 27
(4,415 Views)

I would expect the amount of RAM the OS uses to go up and down (but assume it would be broadly flat over time).

That's the problem: that assumption is not valid on Linux. Linux does things like caching memory in user mode that make it appear to the OS as if memory is being permanently used, when it has been "freed" to the C library but just not returned to the kernel, for example.

As you noted, DSM works around this by reporting not "free physical memory" but "available memory". As I mentioned in my reply to BlueCheese above when he suggested that name, it's a pretty vague name when there are so many possible meanings for "available" with respect to the OS.

It sounds like you would ideally like the current property to be renamed to "available memory" and report the same metric that DSM reports. That "available memory" stat is the result of some not-exactly-self-explanatory math, along the lines of (CommitLimit * 97% - VmSize * 3%) - Committed_AS. The Linux memory manager does not allow non-root processes to use the last 3% of the commit limit, and does not allow a single process to grow too big, requiring it to leave 3% of its size for other processes. All this is also ignoring the complexities of overcommit, which is probably beyond the scope of a short post here but you can read about at https://www.kernel.org/doc/Documentation/vm/overcommit-accounting for details. It also doesn't help that the VmSize * 3% depends on which process you're asking about, and there are many processes possibly of interest in the general case. You can probably imagine why I'm a little reluctant to boil all that down to "available memory", let alone simply replace "free physical memory" with that.

Changing it to report the LV process memory usage would mean it is difficult to know when the program will run out of memory as you don't know how much the OS is using (we often reboot the application if available memory drops beloow a certain value).

That's true and is a valid concern. That's why I wanted to double-check that your question was mostly about leak checking. If you're worried about the entire OS running out of memory (independently of leaks in your LV application) that is yet a different metric.

Is this an accurate summary of your position: the "available memory" heuristic we use in DSM is pretty useful for both leak checking your application and confirming the OS isn't running low on memory. It's not an exact science and that metric technically isn't telling you either of those things to a high degree of accuracy, but it's giving you the basic idea and that's what you're looking for. You'd like it if the existing "free physical memory" stat would report that "available memory" stat instead on Linux. You're not picky about the name or the implementation details as long as the same property can used for application leak checking and OS low memory conditions on all three OS's.

Let me know if I've captured your input accurately and completely. I certainly don't want to make any promises about implementing it, as I have serious qualms about repurposing "free physical memory" to mean anything but literally that, but it's useful input.

0 Kudos
Message 25 of 27
(4,415 Views)

"Is this an accurate summary of your position: the "available memory" heuristic we use in DSM is pretty useful for both leak checking your application and confirming the OS isn't running low on memory. It's not an exact science and that metric technically isn't telling you either of those things to a high degree of accuracy, but it's giving you the basic idea and that's what you're looking for. You'd like it if the existing "free physical memory" stat would report that "available memory" stat instead on Linux. You're not picky about the name or the implementation details as long as the same property can used for application leak checking and OS low memory conditions on all three OS's."

That would be great from our point of view!

I can see the complexities of changing the implementation for that property. Adding an 'available memory' property that reports the same as the DSM would be fine from our point of view too.

Thanks for effort!

Jon

Jon Bowers
LabVIEW Architect, TestStand Architect
0 Kudos
Message 26 of 27
(4,415 Views)

I'm wondering if there has been any resolution to this issue.  If so what is the name of the property that should be used to read a more accurate memory usage stat and what version of LabView is this fix in?

0 Kudos
Message 27 of 27
(4,322 Views)