Showing results for 
Search instead for 
Did you mean: 

Tracking down a periodic delay in a cRIO

Hi Folks,


I am having trouble with a mystery process delaying my main loop in a RT system. I am looking for advice on how to debug this problem. Thanks!


I have an industrial control project in LabVIEW 2009 SP1. It is running in a 9014 cRIO. The main time critical loop runs every 10 ms and both reads and writes using implicit Class 1 Ethernet/IP (Industrial Protocol, a standard from ODVA.) to an Allan Bradely Control Logix PLC.


The PLC has a watchdog counter. It writes a new value on the counter which is then written to the cRIO. The cRIO copies the counter value back to the PLC. The PLC counts how many "cycles" it takes to receive this value back from the cRIO. Each is cycle is 5 milliseconds, the time it takes to run one loop of the ladder logic. If the count of cycles gets to 15, an alarm is set and the control system is effectively shut down.


What I am seeing is that most of the time, the communications works just fine. The data gets through in 2 or 3 cycles, which is what you would expect. About every 6 seconds, the com fails and the watchdog counter spikes past 15. This appears to be something going on in the cRIO.


I did some tests in the cRIO. I set up a timer to tell me how long it was taking to get from the start of the main loop to various points in the loop. The loop is broken up by frames of a sequence structure. What I found was that the loop always ran in the same amount of time, even when the delay was clearly observed. In other words, the delay is happening between iterations of the main loop.


I also tried to use the Remote Execution Trace Toolkit to see what I could see. I set up some code to detect the delay and then trigger a send of the trace data. At best I can get 1.4 miliiseconds of data. Since the delay itself is 90 milliseconds, I am not able to see an event that is relevant to my search.


So how do I figure out what is going on?

0 Kudos
Message 1 of 4

Hello, We had a bit of a strange timing behaviour on our cRIO 9022 that led to the occassional dropped iteration of a timed loop. I spoke to one of the NI Application Engineers who advised that it could be from the Packet Detection setting on the cRIO's ethernet port (configured through MAX). We had it set to the default "Interrupt" and changed it to "Polling" as advised - and we haven't seen the problem since. It sounds like your problem is different, but might be worth checking just in case it is related to this.

Consultant Control Engineer
0 Kudos
Message 2 of 4

I tried setting the port setting to polling at 1 ms. This did not seem to help, but thanks for the pointer. I will put that in my "toolbox" for later.


I do have more info. I tried to set up a Remote trace where the trace would be sent if a long delay in.the Time Critical Loop was detected. So the TCL would write the tick count to a global. A higher priority loop checks at 1 ms intervals if the current tick count is greater then the global tick count by a count of 40. If so then the trace is sent.


I should point out that my loops are all ordinary while loops inside Timed sequence structures. All but one of the loops use the timming primitive that waits for the ms multiple of the number you use as input. (The one that doesn't is a communication loop and has a lot of delays as it waits for TCP/IP comm.)


Anyway, with a timed sequence while loop set for 1 ms waits, the problem goes away.


I am guessing that this loops changes the way tasks are scheduled in the RT and allows whatever task takes over every 6 seconds to be completed over time instead of all at once.


I am not sure how to test this thoery except by changing the loops to timed while loops and setting some of the advanced properties to set scheduling. And that presumes the problem is in my loops and not a process the RT has by itself.

0 Kudos
Message 3 of 4

Not knowing if you are a very experience LV RT programmer, it is difficult to know if the following are obvious to you. The cRIO developers guide is very good at explaining how to handle deterministic and non-deterministic processes, and how you need to decide how to distinguish and implement them and how LV RT prioritises their operation. Chapter 3 is really useful, here are a couple of quotes:


"Your application might consist of both deterministic and nondeterministic tasks. Deterministic processes must complete on time, every time, and therefore need dedicated processor resources to ensure timely completion. Separate the deterministic tasks from all other tasks to ensure that they receive enough processor resources. For example, if a control application acquires data at regular intervals and stores the data on disk, you might need to handle the timing and control of the data acquisition deterministically. However, storing the data on disk is inherently a nondeterministic task because file I/O operations have unpredictable response times that depend on the hardware and the availability of the hardware resource. Therefore, you want to use a Timed Loop for the data acquisition task and a While Loop for data logging. While Loops should be used for nondeterministic tasks or unbounded code (file I/O, network communication, and so on)."




"When a time-critical loop is called, it is the only loop on the block diagram that executes." and "Timed Loops are single-threaded, so you can serialize your tasks using error wires to guarantee that the data nodes execute in the same order every time the loop is called. If you implement these tasks as two separated Timed Loops with equal priority, they still execute in the same thread, but you cannot guarantee the order of execution." and "You generally should use Timed Loops only when implementing a periodic task since any type of blocking introduces jitter."


So perhaps put that non-deterministic communication loop in a while loop rather than a timed structure, otherwise it could be block the other timed loops from running. I'm sure somebody far more experienced with LV RT would come up with a more rigorous recommendation - but hopefully this is a useful pointer.



Consultant Control Engineer
0 Kudos
Message 4 of 4