Out of Memory Error

[bdiesel]

Good Day All,

We have had an out of memory failure on our DCAf application, running on an IC-3120 unit.

The unit is connecting to approximately 30 modbus slaves, some of which were down for a period.

Attached is the Messages log file from /var/log

I am concerned that resources are not being released when modbus slaves are down and dcaf continues re-trying to connect.

Any thoughts?

Bruce

[Jacobson-ni]

Is this a Serial or TCP master? Also, how long is everything up before you get this out of memory error? Is this a slow climb in memory or pretty quick. I know of an issue with the Modbus TCP Slave API which can cause very slow memory leaks but I haven't heard of any issues with the master API.

Edit: Also, what are the error actions or how are you handling the "Unknown" error 56?

[bdiesel]

This is using the dcaf modbus module, as a master.

The leak took about 7 days, which we calculated to be around 0.5GB/day.  Running on a loop time of 200ms

Bruce

[bdiesel]

Attached are the typical error settings for the MB master,  We are using Modbus TCP

[Jacobson-ni]

TCP or Serial? I know there was a recent issue reported with the Serial Master closing sessions incorrectly so it possibly has to do with that.

https://github.com/LabVIEW-DCAF/ModbusModules/issues/59

[Jacobson-ni]

I didn't have an IC immediately available for testing but I did set up a DCAF application on a VM which was configured as a TCP Modbus Master which was connected to a separate VM acting as the modbus slave. This was only one slave but I did connect/disconnect ~200 times with no apparent increase in memory. I also let my Modbus Master sit for another 2 hours or so just trying to connect to a slave which didn't exist but that didn't seem to cause any memory growths easier. For comparison your log only shows 104 disconnects over the entire run and the few I sampled seemed to indicate that the recovery happened within a few seconds.

It could be the modbus module but my testing didn't show anything and I couldn't see anything obvious in any of the module code (I went and matched all of the open references in the init code to close references in the close code). It looks like you have a few custom modules but if you could run this on a PC I would be interested in seeing if DETT picked up anything, if Profile Performance and Memory tool pointed to any VIs growing in size, or if there were a bunch of clones being asynchronously launched and not closed.

[bdiesel]

---

@Jacobson-ni wrote:

I didn't have an IC immediately available for testing but I did set up a DCAF application on a VM which was configured as a TCP Modbus Master which was connected to a separate VM acting as the modbus slave. This was only one slave but I did connect/disconnect ~200 times with no apparent increase in memory. I also let my Modbus Master sit for another 2 hours or so just trying to connect to a slave which didn't exist but that didn't seem to cause any memory growths easier. For comparison your log only shows 104 disconnects over the entire run and the few I sampled seemed to indicate that the recovery happened within a few seconds.

It could be the modbus module but my testing didn't show anything and I couldn't see anything obvious in any of the module code (I went and matched all of the open references in the init code to close references in the close code). It looks like you have a few custom modules but if you could run this on a PC I would be interested in seeing if DETT picked up anything, if Profile Performance and Memory tool pointed to any VIs growing in size, or if there were a bunch of clones being asynchronously launched and not closed.

---

I have set up my dcaf config that has been stripped down to only contain the modbus modules.  Running this on the RIO, with the profiler running, shows all memory stable when all the connections are fine.  Stopping the slaves, causes the "DCAF Execution Service.lvlib:send engine statuses.vi" memory utilisation to start climbing.  Looking at this vi, it is placing messages in a queue.  Not sure if this queue is set up with an infinite queue size, but this queue continues growing.

Note that we are using the dcaf as a service, not the standalone engine.

Bruce

[Jacobson-ni]

Bruce,

I'm not sure if you are able to attach any of your code but I pasted an image of the top level VI in the Execution Service project below. The send engine statuses.vi enqueues messages to the Engine Response while loop in this example so the queue should be limited to 100 items and the message handling loop should just unflatten the message from XML and feed it into the control.

If you are only using this top level VI and haven't changed anything let me know and I can try running this on my end.

[bdiesel]

Hi Matt,

I generated a new Execution Service project with just a couple of tags in it.  Running this continuously on my cRIO, I get a leak of approx 50MB over a 16hour period.  In this instance, there are no modbus connections, or errors being generated.

Attached is the .pcfg file that I am using, a log of memory utilisation on the cRIO, and screenshots of NI-Max at the start of the 16 hours, and at the end.

This is using the default generated Execution Service project.

Bruce

[danijobe]

Did you ever get this resolved?