Ethernet/IP bit collision in PLC?

[Cranky]

TL;DR - If two cRIO chassis write to the same Ethernet/IP tag using implicit messaging, why would the values be OR'd instead of overwritten?

I ran into an odd situation this week. We have two end-of-line testers on an assembly cell that use a NI cRIO-9063. They communicate with an AB Logix PLC over Ethernet/IP using explicit messaging. We were seeing some weird behavior where the typedef'd state machine variable (as seen on the PLC) was showing "impossible" values. We checked the PLC to verify that there wasn't some wayward instruction toggling the mystery bit. We also tried masking the output from the first tester with a value that would prevent the wayward bit from being sent and it still showed up.

After some head scratching and a good night's sleep, something occurred to me: the second nest wasn't being used yet since the product it was supposed to run hasn't launched, but during some troubleshooting the cRIOs were physically swapped. The end result was that the PLC tag names on both cRIOs were writing to the tags assigned to nest one. I.e. "Nest1_State" on both instead of Nest1_State and Nest2_State. The first machine was responding to the PLC as expected (start cycle, change state, etc.) and the other machine was constantly throwing an error during setup, causing it to cycle between the Error (index 😎 and Init (index 0) states. The end result was that the state from nest 2 was sometimes being OR'd with the state from nest 1. Specifically, the Error state on nest 2 and the Idle state (index 1) on nest 1 would show up as index 9 on the PLC, which is a valid state during some portions of the test. Sometimes the overlap was where we expected to see state 9, sometimes it wasn't, but it was there on almost every machine cycle. The weird part is that the only unexpected state we ever saw was 9, which would only be possible if nest 1 was in Idle and nest 2 was in Error, even thought the overlap should have been happening during states 5, 6 and 7.

So yeah, what would cause that to happen? Is it something in the LabVIEW library and how it writes to the data structure? Would it be in the way the PLC receives the network packets? Ideally you should never do this, but we stumbled on to it and it's baffling me.

Thanks!

Cranky