cRIO-9144 (EtherCAT slave) freezes after master's clock changes

[JKSH]

(Copied from http://forums.ni.com/t5/LabVIEW/cRIO-9144-EtherCAT-slave-freezes-after-master-s-clock-changes/m-p/28... )

Hello,

I have these chasses:

• cRIO 9024 (EtherCAT Master)
• 1x NI 9023
• cRIO 9144 (EtherCAT Slave)
• 1x NI 9023

The EtherCAT Master has these software items installed:

I monitor all the system's I/O channels in the Distributed System Manager. I apply an input, and I can see the values update.

However, when I manually change the current time on the cRIO 9024, the EtherCAT slave stops working:

• In the DSM, all of its channel values are completely frozen, while the EtherCAT master's channels still oscillate around the noise floor.
• The "ERR" LED on the cRIO 9144 does not turn on.

(I also tried making the cRIO 9024 a PTP slave, and updated the PTP master clock. The same thing happened -- EtherCAT slave stopped transmitting data)

Is this expected? My full project will involve many CompactRIOs, and I need to keep all their clocks in sync (using something like PTP or NTP). How can I synchronize the different cRIOs without losing EtherCAT slaves?

Thanks in advance

[Catherine-B]

Hi JKSH,

After changing the clock, have you power cycled the chassis?

If you change the configuration for the EtherCAT devices to Configuration mode, then back to Active mode, does the I/O start updating again in Distributed System Manager?

[JKSH]

Hi Catherine,

Thanks for your response.

Power cycling does restore normal functionality, but it's not an feasible solution because we want to make sure that clock updates don't interrupt data acquisition (e.g. if the clock is changed due to an NTP sync event -- CompactRIO clocks have an accuracy of 35-200 ppm which causes significant drift over a month)

I've found the cause of the the freeze: When the clock is changed (either forwards of backwards), the EtherCAT master throws Error -65512 ("The data transfer for some I/O Variables on this target could not be completed in the allotted time and the updates for some values may have been delayed. Increase the Scan Period to avoid this problem.")

In the Scan Engine Fault Configuration page, if I ask the EtherCAT master to treat Error -65512 as "Minor" or "Ignore", the freeze won't occur even after a clock change.

My questions now are:

1. Why did this fault stop acquisition on the EtherCAT slave modules only, while the EtherCAT master modules continued working?
2. I think setting Error -65512 to "Minor" or "Ignore" should be an acceptable workaround, given that the system has already demonstrated that it can handle the current loop rates and I/O channel numbers. Can you think of any other downsides I should consider?

Thanks again for your time!

[Catherine-B]

Hi JKSH,

I spoke with some of our R&D engineers about this. There isn’t a terribly straight forward answer to why the slave modules freeze, while the master does not. It might be because the slave is networked and misses the communication period, while the master does not.

The only downside to ignoring that error is that you lose the ability to monitor it. Sometimes that error can be thrown if something causes the IO to run late.

It is important to note that adjusting the clock should be done during the initialization phase of your program. If you use SNTP to synchronize the cRIOs, it will have a bit of overhead during initialization but then should run in the background and shouldn’t interfere with the EtherCAT communication. If you configure the SNTP synchronization as described in the following article, do you still see the freeze with the EtherCAT slaves?

http://digital.ni.com/public.nsf/allkb/F2B057C72B537EA2862572D100646D43

[JKSH]

Hi Catherine,

Thanks for following up on my questions; I appreciate your help.

I haven't had time to try the steps in the article you suggested, but my colleague had tried it a few months ago to study cRIO clock drift (he thinks it's the same article as what you gave me). He observed that the cRIO clock made a large jump at startup, and then periodically made small jumps automatically (probably to compensate for clock drift).

In my original experiment (the 1st post of this thread), I updated the clock -- both directly to the cRIO, and indirectly via PTP -- to simulate the small jumps caused by SNTP running in the background. My goal was to check if time-sync technologies were compatible with EtherCAT, not to check if we could manually update the clock at runtime. From these observations, I hypothesize that the SNTP jumps would have caused EtherCAT to freeze.

But anyway, I've asked the cRIO to ignore Error -65512, and it has been running fine for 1.5 weeks even with PTP and EtherCAT both active. We now have a way to use time sync with EtherCAT, so we've closed the case.

Thanks again!

[ZhouShanghai]

Hi,

EtherCAT needs the absolute time control, unless one does not require DC feature of EtherCAT.

NI EtherCAT master will take the time control to synchronize the timing between master/cRIO controller and the slaves connected. If there were other timing adjustment components in the system, it is not surprising that EtherCAT network fails to update the data.

Try turn off the DC on all slaves and then you need to adjust the time by other processes/components.