LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

RT jitter! Can multiple reads to a variable / cluster cause a blocking condition?

Solved!
Go to solution

Howdy do.

While incrementally developing and testing an application on a crio9068 (linux RT)  I've begun to see the 'finished late?' indicator in my main timed loop flicker. Starting to pull my hair out trying to figure out how to prevent this from happening. I made a 'hold max' vi and can see the longest execution time for each frame.

 

The application runs fine at about 75% processor load with the front panel open, and the majority of iterations execute in time. Occasionally, I'll have a 'spike' in execution time, and all four frames in the timed loop take significantly longer than normal to execute, and the 'late' indicator says so.

 

A couple questions I've had build up while chasing this:

 

-If I use local varables to pass data between loops, but only write to the variable in one place, can I still cause a blocking condition/jitter by competing reads of that memory space?

 

-If I use an FPGA read/write node to pass data between the timed loop and the FPGA, should I expect this to cause a problem? I selectively disabled a lot of my code, and it seems like this is where some of the delay occurs. What stumps me is that these parts of the code haven't changed in recent development and the thing never used to run late.

 

-On the topic of the FPGA read/write node, I previously assumed that I shouldn't write to the same FPGA FP item in different subvis. However, the code is set up so that there are multiple parallel calls to the read/write node, just with different elements selected. Is this BAD?

 

-Similarly, if I unbundle and read the same element from a cluster control in a 'parallel' fashion, can this cause a blocking situation, or is it the same as unbundling and wiring from there to multple places?

 

-I am using the recently renamed "NI software calibration and management toolkit (SCM)," formerly Drivven CalView, to handle communication between the RT and a windows host. It also does neat fault management stuff. Anybody else using this, and is there any possibility I'm getting jitter by having too many calpoints in my deterministic loop?

 

Any guidance on any of the above points would be greatly appreciated. If I don't make sense on any of the above points I can make example snippets to describe.

0 Kudos
Message 1 of 6
(2,900 Views)

Hello matteci,

 

This doesn't answer all of your questions, but a few points off the top of my head:

 

-Local variables should be relatively safe with regards to jitter, as each creates a unique data copy.

-Timed loops are serialized.  Even if you have code paths that would normally be parallel within these loops, they're serialized at compile time.  Without knowing more about your application I can't say whether or not this will affect you, but it sounds like it might.

-Unbundling or branching clusters shouldn't affect determinism unless the cluster itself contains unbounded data types (strings, variants, variable-size arrays, etc) that are being modified or you're accessing the cluster by reference.

-If you're running this application interactively from the project (which it sounds like you might be, based on the fact that you see the indicator on a loop flicker) keep in mind that this operating mode does not guarantee determinism and adds additional overhead.

 

 

When your loops do run over, how much by?  Have you attempted to cache any runtime data about what the loop was doing or what data it was operating on when it overran?  Have you tried segmenting your loops using additional frames (right click on loop border) to narrow down which operations are causing the hiccough?

 

Regards,

 

Tom L.
Message 2 of 6
(2,852 Views)

Tom,

Thanks for your input(s). I'll stop obsessing over the local variables and the branched cluster wires for now.

 

I didn't realize that all the code in the timed loop would be serialized beyond normal execution. In fact, this brings up another question I have. Somewhere I read that the overhead of multithreading would cause an issue. Since the 9068 has two cores, I had previously been setting the CPU selector in the timed loop to 'automatic', which seemed to load both cores roughly equally. Doesn't this mean that the process is being multithreaded? Funny thing is that even when I do select cpu 0 or cpu 1, they both are roughly equal in utilization while the timed loop is running.

 

The period for the timed loop is set at 15ms, and the execution of all the frames usually occurs in less than 10ms. After several seconds I'll get a 'spike' in execution time, and it will take 20-30ms to complete an iteration. I'm not positive if my benchmark is valid, but if I look at the execution time for each frame and 'hold' the maximum time, it seems like they all (four frames) take extra time at this one instance. So that hasn't helped to narrow it down much. 

 

It sounds like you have a method in mind for 'caching runtime data'. If you can point me in a good direction to gather more information about what the thing is doing it would help. I have run a strip chart of execution times, attached.

 

How much should I expect having the front panel open will affect the determinism of the loop? I realized it added overhead, but since the overall CPU load is less than 60% (each) with all the bells and whistles (other loops) disabled, I thought it wouldn't be having an effect like this.

 

Again, thanks for throwing ideas around, it really helps.

 

Matt

 

 

0 Kudos
Message 3 of 6
(2,844 Views)
Solution
Accepted by topic author matteci

Hi Matt,

 

As far as multithreading- the same thread can be run on multiple cores, but everything in a timed loop will still be scheduled in a single thread.  For reference (see "Defining the Order of Execution for Real-Time Timing VIs"):

 

LabVIEW Help: Timing Deterministic Applications (Real-Time Module)

http://zone.ni.com/reference/en-XX/help/370622L-01/lvrtconcepts/timing_control_loops/

 

For caching the runtime data, I didn't have a specific method in mind, rather I usually try to catch at least one instance of the hiccough by capturing variables of interest in shift registers or an FGV, just to see if there is any sort of discrepancy.  This might not be useful in your case, as I suspect the interactive execution is what's causing the problem.

 

As for the interactive execution and determinism, there's really no way to know what'll happen but it'll often cause the sort of behavior you're seeing now. Because you're more or less streaming data between the target and the host, any hangs in the network or on the host machine can affect the RT system, particularly if you're reading or writing data to the front panel. Since all of your loops are being delayed at the same time this seems the most likely scenario unless you have a non-reentrant VI or similar shared between all loops- I've seen that a few times with shared error or shutdown handlers.  Since these blips are pretty often (it looks like something like 1/100 iterations, I'd recommend putting in a few benchmarking RT FIFO shared variables and deploying the application as a standalone RTEXE.  It should be pretty easy at that point to see if the jitter persists.

 

Best Regards,

Tom L.
Message 4 of 6
(2,837 Views)

Tom,

Yup, it was the interactive execution causing the majority of the issue. When deployed as a standalone RTEXE as you suggest, the jitter is greatly diminished (now execution is between 7-13ms). I also found a couple places in my code where I was using 'build array' every iteration, and preallocating those arrays seems to have helped a little too.

 

Thank you kindly!

 

Matt

Message 5 of 6
(2,793 Views)

No problem- glad to help, and thanks for sharing your additional insight regarding array allocation - that's definitely a good idea and it'll probably help someone else down the road!

 

Regards,

Tom L.
0 Kudos
Message 6 of 6
(2,786 Views)