RAM usage increase while using DQMH on RT target

[mkmarinov]

Hello I've been having this nasty issue with one project for weeks. I'm running a DQMH based project on an CRIO 9063. The controller is not connected to a computer but to a HMI touchscreen via Modbus.

In total there are 17 modules. Most of them periodically read some I/Os and broadcast the data which is sent to a "Main" module that decides what to do based on values. It is mostly sending requests to read the IOs again and sending requests to the "Modbus" module to update values on the screen. In all modules that are requested periodically there is some delay between 100-500ms in order to not have too much communication between modules. The problem is that ram usage of the program increases slowly with time (~20MB per hour) and in few hours the controller goes out of memory and restarts. I checked the queue sizes and it seems that none of them has increasing elements with time. Checked the events queues of the modules via the "Event Inspector Window" although I'm not sure how well it works with RT software. Even tried to flush periodically the user events to see how much of them have piled up. Didn't see any issues there. Also tried the "Profile performance and Memory tool". Although I'm not sure that I used it in the best way it seems that the request event VIs increase their memory usage with time. That seems strange to me but also doesn't seem enough to cause the issue. And lastly the only thing that seems to help is to increase some of the delays that I mentioned before. This helps a little but only decreases the speed of ram filling and I suppose that its just because the request event vis are called less frequently. Any suggestions what could the reason be?

I may not be able to post the entire project but I will try to put together an example to illustrate the architecture better. 

[joerg.hampel]

Seeing as various people (including ourselves) have been using DQMH on Linux RT targets without the problems you describe here, I doubt that we'll find the source/solution in DQMH itself. 

We've seen RAM increases due to the hard drive filling up, due to actual programming bugs (not disposing of references and other resources properly) and due to event structures being blocked from execution (and this event queues filling up).

Evaluating the free amount of RAM on Linux is tricky by itself, btw, and we collected our thoughts on it in our dokuwiki.

Long story short, I'm sorry to say that I can't really think of anything specific to suggest from your description. As always, our DQMH Trusted Advisors would be happy to assist you if we can't get to the bottom of this here in the forums.

[Ozfarmboy]

1) I haven't used it much myself, but there is the RT Trace Viewer which you can use to help debug your RT application.  Have you given that a go.  Tools --> Real-time Module --> Trace Viewer.

2) Another consideration - check that you don't have string/array controls on any of your front panels that are constantly growing.  For instance, if you have a Status Indicator (string) on one of your front panels that is constantly displaying broadcasts such as the Status Updated broadcast, this will constantly grow over time.  With these type of indicators, we limit the size to say 100,000 characters so that it doesn't require endless memory.  Same goes with arrays.  You can also pre-allocate these things (ie. Initialise array with a specific size, and taking this approach makes your code more deterministic.

3) A third consideration - run the VI Analyser on your project and see if it picks up anything.  The VI Analyser runs a lot of different tests, and might highlight something you weren't aware of and will provide a recommendation to resolve.

[joerg.hampel]

---

@Ozfarmboy wrote:

1) I haven't used it much myself, but there is the RT Trace Viewer which you can use to help debug your RT application.  Have you given that a go.  Tools --> Real-time Module --> Trace Viewer.

---

I have never been able to get any useful information from the Real-Time Execution TraceToolkit when debugging memory-related problems. Yes, the tool can show memory management operations for each thread. I could never use this to my advantage, though.

[Ozfarmboy]

I think they've renamed the Real-Time Execution Trace Toolkit to RT Trace Viewer.

[mkmarinov]

Hi again! I also tried to use the RT trace tool but couldn't figure out how to do it properly.

I also just ran the VI analyzer to see if something comes out. I ran it only on the "Main.vi" of each module and some custom SubVIs used since I assume the problem is not in the DQMH itself. Unfortunately I found nothing suspicious. I'm uploading the report just in case. 

[mkmarinov]

Hi again everyone! I managed to strip down the project to the very essentials. The problem still persists. I guess that's good  because now it will be easier to pin point the root cause. I'm uploading the project if someone want's to take a look. 

[Ozfarmboy]

Hi.

I had a real quick look. 

A couple of suggestions:

1) I noticed you're constantly closing the shared variable after reading/writing to it.  Maybe this is your issue.    Try opening/connecting to the shared variables on startup, and write/read as you go, and then close them all at the end.  Maybe you're constantly having to allocate new memory each time and that's not being handled well.

2) I recommend running each module via the tester on its own and see if you can isolate the memory issue to a specific module.  I'd start with Variable Wrapper module, and then progress to wherever you're calling the shared variables.

If this doesn't solve it, I think you're at the point where it might be worth your while consulting with a DQMH Trusted Advisor.  Although I don't think the issue is actually DQMH related.

[mkmarinov]

---

@Ozfarmboy wrote:

Hi.

I had a real quick look. 

A couple of suggestions:

1) I noticed you're constantly closing the shared variable after reading/writing to it.  Maybe this is your issue.    Try opening/connecting to the shared variables on startup, and write/read as you go, and then close them all at the end.  Maybe you're constantly having to allocate new memory each time and that's not being handled well.

2) I recommend running each module via the tester on its own and see if you can isolate the memory issue to a specific module.  I'd start with Variable Wrapper module, and then progress to wherever you're calling the shared variables.

If this doesn't solve it, I think you're at the point where it might be worth your while consulting with a DQMH Trusted Advisor.  Although I don't think the issue is actually DQMH related.

---

Hi! I actually I added closing the shared variables recently because I thought the issue is they stay opened after each read/write. But it seems that it doesn't matter. 

[Ozfarmboy]

You can use Open Variable Connection.vi to open the variable once upon startup:

I guess the question is why you use the explicit VIs like this rather than just placing a shared variable on the block diagram?