Deploy and run at Startup for RT systems

[henriksandin]

Hi
We're using the DQMH architecture on our cRIOs and we've run into a problem.   

We have one instance of DQMH that is running as a clone. This works great and our top level application can spin up multiple copies of the module and all is good. We have now built and deployed this top level application onto the cRIO as an RT application and it is set to run at startup. This works great. 

But now we have a problem. We now have a singleton DQMH module. We have tested it in debug mode by running it from our develop computer and it works great. But when we build the application and run it as an RT application deployed to the RT system it stop working. We get sync errors and any controls we try to set at startup (in the Module startup VI) does not work. No Controls are set in the module itself after startup.

The only difference is the "Open VI reference VI" for the two (And yes running the clone module as a singleton give the same error as the singleton itself). 

Can you think of any bug that would do this? Any thoughts?

We are running our code on the cRIO 9038

Cheers

Henrik

[FabiolaDelaCueva]

Hi Henrik,

We are waiting to get one of our Real Time systems up and running to try this out. 

In the mean time, here are a couple of thoughts:

1) To rule out DQMH code, I would create a simple VI that gets Open by reference and use the Run VI server. If the issue is reproducible with this simple VI, then it is a LabVIEW bug.

2) The Real Time OS is supposed to be completely up and running by the time the startup application starts. However, in some of the applications we have ran in the past, specially the ones that control hardware, we have learned that it is a good idea to put some delay before starting the application. This could be a startup VI that waits a couple of seconds before launching the other VIs.

We will let you know what we find on our end when we get the Real Time system.

Please let us know if you find anything on your end.

Thanks,

Fab 

[henriksandin]

Hey Fab
We did do a check with just the basic LabVIEW VI's and we do have the same problem. Our local NI reps know about it and will see what the RT people can come up with for this. When we know what they say I'll let you know since DQMH uses this way of starting the modules. Let's see if there's a workaround or if it is something that NI will fix. 

I will for sure try the wait and see if it helps! 

Cheers

Henrik

[FabiolaDelaCueva]

Hi Henrik,

Thanks for keeping us posted. Your local NI rep contacted us last week and we discussed some options as well. I have sent them a CAR ID that might be related and I have reached out to other folks at NI to see if I can gather a little bit more information. 

I believe the workaround for now, would be to not program DQMH singletons, instead use only cloneable modules. You will not be able to use the run as singleton option though, because it seems that the code works when the VI is prepared as a reentrant VI. So you will need to use the Clone ID to ensure all communications with the single instance of the cloneable VI are done with that cloneable VI. You will also need to add extra care as to not call the Start Module in more than one place, because you would want to always communicate with the same instance and not create new clones accidentally. 

I hope this helps. Keep us posted and we will keep you posted of what we find on our end.

Best regards,

Fab

[henriksandin]

Hey Fab

That's exactly our workaround right now. I'm glad we picked the right work around according to Fab! 🙂 

Will keep you posted and thank you for your help! 

Cheers

Henrik

[ChrisStrykesAgain]

Henrik,

Fab reached out to me regarding this and I wanted to check in. To help narrow in on a few things, do you have embedded UI enabled on your target? If so have you/can you try it with it disabled?

I ask because there is a CAR, 514879, that describes behavior that might be similar (A VI Server call not working properly) but only manifests with embedded UI enabled. If you are using embedded UI, let me know and I'll dig further to see if the CAR is actually related in terms of cause and then we can see if the same workarounds may be valid.

[ManuelSebald]

Hi everybody,

I´m the new colleague of Joerg Hampel (some of you might know him) and with this post I´m starting into the LabVIEW community .

We run into the same problem with DQMH on a cRIO. To solve the problem for the singleton modules (we don't want to use clonable modules because of the overhead with the ID) Joerg made a simple workaround:

In the 'Get Module Main VI Information.vi' we mark the vi-reference as reentrant. 

We are aware of the problem with this workaround, that it´s not possible to check if the VI is already running or not.

To get a little more insights, we run a set of tests with DQMH in a simple project (one module) on several RT-targets. In a short summary (I don't want to post big tables here ): 

• NI-cRIO 9030 (LinuxRT), with / without embedded UI, dev / startup mode, with / without workaround
• NI-sbRIO 9636 (LinuxRT), no embedded UI, dev / startup mode, with / without workaround
• NI-sbRIO 9637 (VxWorks), no embedded UI, dev / startup mode, with / without workaround

Used version:

• LabVIEW 2016 (16.0f2)
• NI-cRIO 9030: NI Linux Real-Time x64 4.1.15-rt17-4.0.0f0, Firmware: 4.0.0f0
• NI-sbRIO 9637: NI Linux Real-Time ARMv7-A 4.1.15-rt17-ni-4.0.0f1, Firmware: 3.0.0f0
• sbRIO 9636: N/A

Results: DQMH (without any workaround) works on all systems and in all modes, with one exception. It doesn't work on the cRIO system with embedded UI enabled and in startup mode (it makes also no difference if a display is connected or not). Our tests indicate that the bug occurs only if the embedded UI is enabled. If the embedded UI is disabled or there is no embedded UI on the device, DQMH works fine on RT systems.

Hope these informations will help you.

Regards,

Manu

[FabiolaDelaCueva]

---

@ManuelSebald wrote:

Hi everybody,

I´m the new colleague of Joerg Hampel (some of you might know him) and with this post I´m starting into the LabVIEW community .

Welcome to the community! I am honored that your first post is regarding DQMH 🙂

---

@ManuelSebald wrote:

We are aware of the problem with this workaround, that it´s not possible to check if the VI is already running or not.

The problem with this workaround is more that Singleton DQMH modules are designed in a way to make it possible to call the "Start Module" in multiple places in the code and always return the same instance of the DQMH Main. With this approach, you need to ensure that you are calling Start Module in a single place only and that if you need to restart the module, that you re-register for the module's events.

If you have not done so, I think you should make your modifications part of a DQMH Template. That way you don't have to make them every time you create a new module, here is more information on how to do that: http://delacor.com/documentation/dqmh-html/DQMHDocumentation.html?AddingaNewDQMHModulefromaCustomT.h...

---

@ManuelSebald wrote:

Results: DQMH (without any workaround) works on all systems and in all modes, with one exception. It doesn't work on the cRIO system with embedded UI enabled and in startup mode (it makes also no difference if a display is connected or not). Our tests indicate that the bug occurs only if the embedded UI is enabled. If the embedded UI is disabled or there is no embedded UI on the device, DQMH works fine on RT systems.

Hope these informations will help you.

Regards,

Manu

---

Thanks for the extra details and information. We are following up with LabVIEW R&D to see how else we can address the issues you are seeing when using embedded UI. These extra details will be very useful. 

We have also filed this issue in our internal issue tracking system and we will consider it for future DQMH releases. Specially if we decide to create a separate DQMH Real Time template.

Best regards,

Fab

[FabiolaDelaCueva]

---

ManuelSebald wrote:

Results: DQMH (without any workaround) works on all systems and in all modes, with one exception. It doesn't work on the cRIO system with embedded UI enabled and in startup mode (it makes also no difference if a display is connected or not). 

---

Manu,

I have been thinking about this more. You say "in startup mode", which is similar to what the original reports talked about. 

I have seen weird things happening when running a VI as a startup in a Real Time system. The Operating System is supposed to have loaded all we need to run our startup application. However, in the past, what I have done is to create a sort of splash screen VI. The startup VI is just a VI that has a delay of about 5 seconds and then that VI launches the VI that I actually want to run as a startup. I have not figured out why this hack works, but it has saved me multiple times when we have worked with motion projects in Real Time.

If you have a chance, can you try running a VI that has the 5 seconds delay before starting your DQMH module? Of course, this would be with the Start Module without the workaround. Does this fix the issue?

I will try to get a Real Time system to try on my side, but thought you might have something available. 

Thanks,

Fab

[joerg.hampel]

---

@FabiolaDelaCueva wrote:

The problem with this workaround is more that Singleton DQMH modules are designed in a way to make it possible to call the "Start Module" in multiple places in the code and always return the same instance of the DQMH Main. With this approach, you need to ensure that you are calling Start Module in a single place only and that if you need to restart the module, that you re-register for the module's events.

Fab, the two of us already discussed this at NI Week, but I want to repeat it here to keep this discussion complete: The architecture of those real-time applications that we build on DQMH is designed in a way that only allows on specific place for creating modules. We do not start or restart DQMH modules based on events etc. at this time.

If you have not done so, I think you should make your modifications part of a DQMH Template. That way you don't have to make them every time you create a new module, here is more information on how to do that: http://delacor.com/documentation/dqmh-html/DQMHDocumentation.html?AddingaNewDQMHModulefromaCustomT.h...

We already did that, although it didn't work out completely in the end. We will get back to you about that and create another topic for that.