Issue with Target RAM Ramping Up on sbRIO & myRIO

[pegahm]

Hi,

I am running an application on 6 different NI targets for 2-3 months in the form of an endurance test to validate my application on multiple targets over a period of time. Targets are either myRIO 1900 or sbRIO 9637. I have observed that sometimes, all of a sudden the RAM on a target starts to rise up until that target crashes. All of the targets run the same application; therefore, i was wondering if it could be as a result of something happening beyond my application and maybe something happens with Linux RT. My application is a relatively large application including CVT operations, launching subVIs, using TDMS at 10H, etc. The is reason I became suspension about something else beyond my application, is that it only happens to one or two of the targets after I start them all at the same time. It doesn't happen to the same target all the time though. 

I was wondering if this has been seen before by anyone? or if anyone would know about possible causes of RAM ramping up suddenly if target has been running for an extended period of time?

Thank you,

Pegah

[wiebe@CARYA]

Are you sure it only happens on one of the targets (which one btw)?

My bet would be that it crashes on the smallest target first. sbRIO 9637 has 512 MB RAM, the myRIO 1900 has 256 MB. If 206 MB is used from the start, that would take the 9637 5X longer to crash!

Are you sure (how do you know?) the memory increase is not from the start?

---

@pegahm wrote: 

or if anyone would know about possible causes of RAM ramping up suddenly if target has been running for an extended period of time?

---

Possible causes? My bet would be your code, maybe combined with some external trigger (connection lost, connection created, could be anything). Otherwise it could be a very very obscure bug.

Any way to speed up the crash? Try stress testing: increase loop speeds and hope you see the crash more often. That would be by first priority: speeding up the crash so it happens more often. Then prevent it, then slow down again.

[pegahm]

Yes, I am sure that it only happens on few of them and in a random manner, meaning that it happens that sbRIO carshes and myRIO is still running. The endurance tests report every 12 hrs so I monitor the RAM on each target on frequent basis and I exactly know which one crashes.

The external trigger that you mentioned might happen in our system. Could you elaborate on that please. Is there a known fact that external triggers may result in target become "unstable" all of a sudden until it crashes?

[algarcia]

I think he was referring to the fact that the external trigger might not being received by the board because the CPU might be hogging all of the resources.