I have what I think is a TSN sync issue between 3 targets each running a couple of DAQmx tasks. I have seen a couple of different error messages come up.
The system consists of two 9045 and one 9049 cRIO's in a star network pattern with the Cisco IE4000 switch. At this point I'm only interested in time sync...deterministic messaging comes later so I haven't even touched it. The switch was configured to enable timesync but nothing more.
Sometimes I'm lucky and everyone syncs and the tasks all work, but most of the time one or more of my tasks errors with one of the following errors:
Error -209860 occurred at an unidentified location
One or more devices could not be added to your task because they have existing coherency requirements that conflict with the new requirements of your current task. Stop the task(s) that use devices in this task and try running it again.
Error -200284 occurred at an unidentified location
Some or all of the samples requested have not yet been acquired.
To wait for the samples to become available use a longer read timeout or read later in your program. To make the samples available sooner, increase the sample rate. If your task uses a start trigger, make sure that your start trigger is configured correctly. It is also possible that you configured the task for external timing, and no clock was supplied. If this is the case, supply an external clock.
Error -209836 occurred at an unidentified location
The devices in your task cannot be synchronized. This may be because there are no available synchronization mechanisms between the devices.
Some synchronization paths are not available in interactive tools like the DAQ Assistant. To determine whether synchronization between these devices is possible, try deploying and executing your task in your application environment.
I'm new to TSN, but these errors lead me to believe that onboard clocks are not syncing with the grandmaster to within the allowable propagation delay. On app startup, I threw in the 802.1AS status vi and I can see from my logs that every time a cRIO powers up it becomes the MASTER. When I deploy new SW to the targets (all running same rtexe, so I always deploy to all 3 at once), they restart at the same time but of course the order with which the app actually starts is sorta random. I can look at the logs and see that most of the time when I check the status that target is the master. I need to add recurring checks to watch for changes in master and read the delay...that's probably next.
I can run the same rtexe on a cRIO running 7 different DAQmx tasks and it has no issues when it's all intra-chassis. Only when trying to sync across TSN.
I briefly dabbled with setting the propagation delay using nisync prop nodes but I got the coherency error still (I set it all the way up to 5000ns from the default of 800ns).
Sorry, but I'm not able to share this code.
Some suggestions on debugging process. I would recommend you start with debugging and assuring proper operation of time sync before moving on to the DAQmx errors. Time sync is a separate driver and process. Once you know that is working properly it may have already fixed the DAQ issue or at least you'll know you are working from a good starting point.
If you run the .1AS monitor on all three devices (or monitor remotely) do you see that they all have the same GM? One may be a master but the other two should be slave (or all three could be slave and Cisco is serving as the master.
Yes, then move on to DAQmx.
If this is not working connect the .1AS enabled ports of two devices directly together and check if they are sync'd (you will need to connect to the device with the other port or over USB to check the .1AS status).
If that is not working then you need to check the installation and configuration of .1AS on the cRIO systems.
If that does work then you need to check the switch configuration to make sure that .1AS is selected (it also supports other PTP protocols).
Let us know what you find and we can help with next potential debug steps.
As a note, you should not need to change the propagation delay. It sets how much delay the physical cable can introduce before declaring a port non-AS capable. It's intended to detect if non-AS switches are used and protect against bad network time sync. The only reason you would need to change is if you were using a fiber converter (will add latency) or if you have a prototype AS devices that is not yet fully functional so you need to disable the check.
A coworker and I recreated the problem in the lab with two 9045's directly connected to each other (at least one of the errors).
A simple vi with a DAQmx task and read was run on each target. The .1AS status vi was also placed in the read loop to see live view of GM, state, and offset. Here is a sequence of events that triggers the error. Note that at the start of these events, the targets were already running. Second note is that one of these cRIO's in this test is connected on the second Ethernet port. I set the status vi time ref ID to be that port so I get valid states and GM's.
This particular error occurs when the GM changes due to a new target entering the TSN system that's higher in the GM pecking order. This seems to parallel what I see on the 3x cRIO system (although I get additional error types on that system that I've not yet been able to recreate in the lab).
I'm going to add .1AS periodic status checking on my deployed system of 3 cRIO's hopefully today. That system is in activation so I have to wait for a good time for the systems to be free. I do know that the GM jumps around when I restart those 3 targets...
I checked the Cisco switch, it's PTP profile is set to 802.1AS and is enabled on all ports.
How would I check the installation and configuration of .1AS on my targets? These are all pretty new devices with nisync and tsn installed via MAX.
Thanks for the note on the propagation delay. Still trying to figure out all these knobs that can be turned for TSN.
From what you have described it sounds like the time synchronization features are all working properly. I think at this point we need someone with knowledge of what DAQmx is doing under the hood and the details of that error to guide. I think they monitor this forum but you may also want to open a support ticket.