RT cRIO: Shared Variable Engine Errors

[Jed_Davidow]

This is an app written in LV2011 running on a cRIO-9076 (Scan mode), with the August release of software installed on it.

I have gown to great lengths to try an optomize my code, I have reached an impasse.  The VERY FIRST TIME I run the code on the target (be it via a self-starting EXE or targeting from my Windows development machine I am able to read and write to several shared variables, but then when I access them again (possibly when I am starting a couple timed loops and the processor is very high) I get this:

Error -1950679023 occurred at Shared Variable in targetIdleLoop.vi->ZDAQ_cRIO_Target_Main.vi
Possible reason(s):
LabVIEW:  (Hex 0x8BBB0011) The connection to the server was disconnected.
This error or warning occurred while writing the following Shared Variable:\\ZeeAero-RIO1\variables - network - RT (separate)\RTStatus\\10.10.10.10\variables - network - RT (separate)\RTStatus

If I go back to the Dev machine and restart the code, I don't get this error; it only happens after a reboot.

BUT on this second run (or subsequent VI starts), when we get to that same point, a client priogram running on Windows and reading a network variable successfully suddenly gets:

LabVIEW:  -1950679035 (Hex 0x8BBB0005) Unable to locate variable in the Shared Variable Engine.  Deployment of this variable may have failed.

Which lasts a minute or so, then goes away.  The code is made to loop, and when it loops back through this point there are no errors at all.

So, to recap:

1st Run after reeboot, 1st loop :Error on Target (0x8BBB0011), connection to server disconnected

Subsequent Runs, 1st loop: Error on Windows Client (0x8BBB0005) Unable to locate variable in the Shared Variable Engine.  Deployment of this variable may have failed.

Subsequent Runs, 2nd+ loop: No error

Here is a link to a screenshot of the installed SW on the Target:

EDIT:  I should mention that there are three loops running in parallel on this app; 1 is a Timed-loop (at the moment) set to a period of 10ms with 100 priority, #2 is also a timed loop and is set to 3ms with priority = 50 and the third is a WHILE loop with a 50ms wait.  The error occurs during the WHILE init phase, which happens once per run, not repeatedly.

[Jed_Davidow]

This is killing me.  It's obviously something to do with when the Timed loops start AND when I access the network variables.

See the screenshot below.

Note that I read and write to several network variables in Subroutines to the left of this screenshot at startup.  The starting of the timed loops causes the issue with the shared variables.  (also note that the periods are set to 100ms and 99ms in my testing after the initial 1st loop.

Now look at the Wait at the bottom set to 10 seconds.  In this case, it will work.  There is no error at breakpoint 37 if I wait around 10s.  If I DON'T wait the 10 seconds, there is an error after the check on the BOOL "stop - Network" variable that NEVER GOES AWAY.  If I look at the CPU monitor, there is an initial bump in the cpu when starting the loops, and then it drops to about 15%.  But if I don't have that 10s wait, there is ERR 0x8BBB0011 and the CPU spikes to 75% every 5 seconds and it never goes away.

And that's only the 1st loop after reboot.  The 10s makes this work, but I still have an issue with it once I start doing individual runs (loops).

[Jed_Davidow]

This is just a waste of my time.  I am chasing down the issues one by one.  I finally REMOVED the shared variables from inside the loops- I stopped posting data and status, AND STILL the client program suddenly cannot contact the Shared variable server during just that first iteration, for 10-60 seconds.  So it's probably going down, even though CPU is not too high and there is plenty of memnory available.

I give up.  A tremendous waste of my time.  UDP/TCP may take a little more effort, but this was goddam WORKING until suddenly something I did changed timing or processor and now this is garbage.  This is why I held off using shared variables so long.  Because I need to use Voodoo to make them work.  Crap.

[Jed_Davidow · Accepted solution]

After a whole bunch of time working on this thing, I have come to the conclusion that if you start to deal with the Shared Variable server too soon after bootingt he target, you get errors.  And if you get those errors, you just need MORE time before things settle down.

I added a loop to wait until the target has been up and running for 90 seconds (60 seemed to work, but who knows?) before I launch into the guts of the program.

[Stijn_Schacht]

Hi,

this is "expected" behavior and I've come accross it in several application running RT startup apps on cRIO's and PXI's. The thing is that the shared variable server is a thread running parallel with your application. When the RTOS boots, it immediately starts your app but didn't yet launched the shared variable service. Therefore, your app runs into problems since it can't find the shared variable processes.

Create a while loop that polls for your error code (its a specific error code, don't know it by heart) until this error dissapears. Then continue with the rest of your program.

Works every time for me, good luck!

[Jed_Davidow]

Thanks for the message (sounds like you dealt with the same issue).

I did what you suggested (polling for the error to disappear.  Problem was that my polling period was too fast; pinging the engine and getting the error extended the amount of time needed to recover.  So just the act of searching for the error causes you to have to wait longer; which in our case (a system that needs to be up and running as soon as possible) was detrimental.

I found a wait of 30-45 seconds necessary.  This was too long for my client, who had be remove the shared variable and use UDP. 

[Jed_Davidow]

It also did not help that this was not documented.

[Stijn_Schacht]

Sorry to hear that... However, it got solved now?

[Jed_Davidow]

Well, insomuch that I removed the shared variable engine, yes.  It's solved!

[Stijn_Schacht]

Hi Jed,

sorry to hear that. The shared variable engines actually do work fine but since LV2011 they changed quite a lot in the way RT systems behave. The speed is definitly improved but the way you - as programmer - must handle everything, requires a bit more in depth knowledge from time to time.

In the future, I would advice you to keep trying with SVe's since you have the luxury of using the Distributed System Manager to debug your application rather than having to write your own client to poll the variables values.

best of luck!

Stijn