Shared variable not reliable? Problems syncing multiple machines

[flowschi]

Hello,

I am having some trouble with shared variables.

We need to sync 6 PCs. PC 1 is the "master" which is deploying shared variables, and the 5 slaves are reading/writing the shared variables - each in it's own section. For example there is a life-bit beeing toggled between master and every slave.

Sometimes it happens that the setup is loosing it's sync, e.g. lifebit times out. The master is reading "false", the client is reading "true" - both without errors. Also other bits are not transferred. In this case, when the slave is restarted, it's working again in most cases.

The SV are accessed programatically using a static IP. The SV is openend, and in case of an error it's closed and re-opened. The refnum stays open all the time. But there is no error - the SV just seems to be not reliable.

In total there are 30 SVs, some of them are just bits, others are clusters. They are all written/read in a cycle of ~500 ms on both sides.

Each PC has a static IP, they are connected through a single switch. The switch is part of the (quite big) company network, so that the PCs are also accessable from outside.

Is it possible that the SVE can be responsible for this problems? Are there any options to configure the SVE for our needs?
Because of the problems we are thinking about swithing to STM.

Thank you.

[crossrulz]

---

@flowschi wrote:

Because of the problems we are thinking about swithing to STM.

---

You might want to look at using Network Streams as well.  Think of them as network queues.  They are excellent for 1:1 communication.

[PatrickLye]

1. Are all of the machines on the same subnet?
2. Do you have good gigabit connections on all machines?
3. Download WireShark and look at your network traffic to see if you are having packet crashes.
4. Try Putting the machines all on a separate switch that is their own network for testing and see if the problems go away. If it does you may be able to set up a virtual network in the switch for these machines and then only allow traffic in/out of the network that is needed.
5. Can you turn off the firewalls for testing?
6. Try turning off logging in your network variable properties.
7. Turn on buffering and set up a buffer size that will work on your network variable properties.
8. Disable the enable enhanced DSC run time support in the Advanced tab of the build executable.
9. I’ve found shared variables to be incredibly dependable on my industrial network and a bit problematic on our office network. They need to have good connectivity, and our office network isn’t all that good. They are working on it.

[Bob_Schor]

I agree with Crossrulz about the reliability of Network Streams.  In a Real-Time project, I was using 4 Network Streams between a PC and a PXI Chassis, one of them streaming multiple channels (16-24) of data at 1KHz to the PC.  We originally had a Shared Variable that we used to send "monitoring" information at a few Hz to the PC, but it proved to be unreliable compared to our Network Stream traffic (of course, part of the problem could have just been the amount of TCP/IP traffic flowing between these two systems).  Unlike Patrick, I've found Shared Variables to be problematic ...

Bob Schor

[MichaelBalzer]

In my experience SVs have generally been pretty reliable, except when they aren't.

The most interesting solution to a SV failing to update I've found wasn't in code, but by running the Distributed System Manager. With the DSM running and the libraries with the problematic variables expanded, the variables updated without a hitch. If the DSM was closed or the libraries not expanded, the code would be writing to the variable, but the updated value would never get read on other machines (cRIOs). Very strange.

Have you tried disabling firewalls on both the server and client PCs?

[flowschi]

Thanks all for your answers so far!

---

@PatrickLye wrote:

1. Are all of the machines on the same subnet?
2. Do you have good gigabit connections on all machines?
3. Download WireShark and look at your network traffic to see if you are having packet crashes.
4. Try Putting the machines all on a separate switch that is their own network for testing and see if the problems go away. If it does you may be able to set up a virtual network in the switch for these machines and then only allow traffic in/out of the network that is needed.
5. Can you turn off the firewalls for testing?
6. Try turning off logging in your network variable properties.
7. Turn on buffering and set up a buffer size that will work on your network variable properties.
8. Disable the enable enhanced DSC run time support in the Advanced tab of the build executable.
9. I’ve found shared variables to be incredibly dependable on my industrial network and a bit problematic on our office network. They need to have good connectivity, and our office network isn’t all that good. They are working on it.

---

1. Yes

2. It's gigabit, bit if good or not - I don't know. They are all on the same switch. But some other traffic is also there (remote access, database connection...) and the IPs are accessible in the whole company (~3000 employees in 4 countries).

3. I tried wireshark in the past, but I don't know how to use it properly to detect network issues. Maybe I can get some support here.

4. We would need an USB->Ethernet adapter, because there is no free ethernet port on the machines. Because we don't really trust these USB devices, we didn't try yet. We'll try it in the next days!

5. Firewall already is turned off, antivirus has been disabled

6. I don't have an "Logging" section. Means I don't have DSC installed?

7. Will buffering improve the "reliability"? If we just get old buffered values, the alive-bit also might be missing.

8. We don't have any EXE running yet. All stations running in IDE. Is it a disadvantage?

---

I read about a logos.ini, which can be used in LV 8 to restrict shared variables to a specific MAC. Is this also possible in LV 2014?

/EDIT: Is seems that the problems are not totally random. The line might run without any issue for some hours. Once the problems start, they will occur frequently. Closing LabView and restarting it on all stations seem to clear the issues for some time. Maybe the SVE will run into problems after some time? Originally we just jused Read/Write shared variable with a connection-string wired to it. Then we started to open the SVs once and then use the refnums - but it did not really improve the situation. But like I said, the read/write VIs do not throw an error - the values are somehow just not equal on both sides.

Is it possible that the network (switch?) can run into problems after some time?

[Sam_Sharp]

---

@flowschi wrote:

But like I said, the read/write VIs do not throw an error - the values are somehow just not equal on both sides.

Is it possible that the network (switch?) can run into problems after some time?

---

The shared variable engine works such that every client has it's own local copy of the shared variable engine, when you write to the shared variable it gets posted to the local instance and the values are only pushed to the network (i.e. other clients) periodically.

There is a lot of information about how they work and the options available on the NI website if you do a search e.g.: 

http://www.ni.com/white-paper/4679/en/

http://www.ni.com/white-paper/5484/en/

[flowschi]

Hello,

thanks for your support so far.

My updates so far:

Last friday we added an USB network interface to the 5 PCs in this line.

We already have 2 network cards on every machine, but the SVE was runnnig together with other services.

Now we have it running on the seperate USB network cards, which are connected through an seperate switch (static IPs).

Unfortunately we still have the same problems. It seems that nothing has changed so far. 😞

I also made use of the logos.ini, even if I am not sure if it still applies to LV 2014: http://digital.ni.com/public.nsf/allkb/D358BABAE2F61F1A862571570056867D

Unfortunately I don't think it's working:

Using my notebook connected to the company network, I can still see the shared variables using the IP of the 1st network card of the PC deploying the SVs.

Means: The SVE is bound to network card 3 using logos.ini, but network card 1 still seems to deploy the shared variables.

Might there be a problem because the shared variable engine has multiple network cards in use?

I think we'll switch to network streams, but it will take some days until I have the time for it.