1588 on cRIO 9056 is corrupting DAQmx HW timed AI

[viScience]

When 1588 timesync is running it appear to interfere with DAQmx AI timed tasks.  I am seeing strange errors

-200284 (buffer underrun) or -209836 (The devices in your task cannot be synchronized) when I start my DAQmx task with 1588 engaged. When I disable the 1553 master on my network or remove the 1588 driver from the cRIO everything works correctly.  To simply testing I am using a DAQmx example - Thermocouple - Continuous Input.vi to do my testing. I do not have any HW sync modules just plain C series AI modules so there should be no attempt to perform any AI clock slaving since this is a software only 1588 system.  BTW I am using the latest 19.0 drivers and LV Dev.  I also see the same behavior in 18.5.

[cweeks12]

Hi,

I found this article referring to the error you're seeing, which suggests compatibility may be to blame with this: https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z0000015CisSAE&l=en-US

[viScience]

I have tried this with the newest recommended v19.0 and the previous v18.5 installation sets.  I see the problem in both cases. 

There is no way that 1588 sync should be touching the AI timing signals unless you have HW sync capability which I do not have in my system.

[esattern · Accepted solution]

The cRIO-9056 does have hardware sync capability. It runs 1588 with hardware timestamping enabled and will try to align all of the clocks on the device (including the sample clock for DAQmx tasks) to match the grandmaster's time. If it's slaving to a software-based grandmaster, that could explain the synchronization error you're seeing. Software-based references usually have a lot of jitter in their ability to deliver time which could easily cause your cRIO-9056 to start railing the oscillator's frequency up and down to try to match (and/or introduce discreet steps in time to catch up). If you want to confirm this behavior, you could use the NI-Sync API to monitor the offset between your cRIO and your grandmaster. The "Monitor Synchronization.vi" example demonstrates how to do that. To get the best performance, you'll need to pick a grandmaster for your network that supports hardware timestamping (perhaps the cRIO would work for your case?). Alternatively, if the performance degradation is acceptable for your use case, DAQmx tasks have an option to ignore sync errors, I think it's called "Ignore Sync Lock Loss". If you turn that off, you may want to add some logic to your application similar to that example I mentioned so you don't miss any potentially large losses of synchronization (like the grandmaster going away).

[viScience]

Well that seems to be consistent with what I am observing but what about us folks who just want our cRIO system clock to be sync'd?  There should be someway to disable these advanced TSN features.

[viScience]

Yep that works.  I would never have found that obscure property without your help, thanks!

[viScience]

Well it seemed to be working but the other day the error popped up again so I guess there are some edge cases where this problem can still happen.  I was able to get it working again by shutting down all of the cRIO's and the Master PC based time server then restarting a single cRIO and the time master.  I then was able to bring the rest of the controllers back online and the system seems stable for now but there is certainly a potential for this to happen again.  I think the correct solution would be to find a way to disable HW AI sample clock sync, any ideas?

[esattern]

Which error did you get? If it was, -209836 (The devices in your task cannot be synchronized), double check that the "Ignore Sync Lock Loss" property was set. You shouldn't be seeing that particular error if that property is set. I'm not too familiar with -200284 (buffer underrun) so someone who knows DAQmx better might be able to speak to that case.

Regarding disabling the hw AI clock synchronization: Unfortunately, I don't believe there's a way to do that at the moment. Maybe we can work around it, though. The most straight-forward option would be to pick a better quality 1588 grandmaster. Perhaps you could set the cRIO-9056 to be the grandmaster and your Master PC could act as a boundary clock to all the other devices on the network? If that's not an option, how accurate do your sample timestamps have to be relative to everything else in your 1588 network? There may be a solution we can find that doesn't involve running 1588 on the cRIO - e.g. by correlating the cRIO's system time to the 1588 network through some other mechanism.