Windows OS determinism problem

[chris_1243]

I have a While Loop in my program that I need to execute under 10ms on every iteration. I have removed all subVIs from the loop and I have a 5ms Wait. Normally, the While Loop iternates within the 5 to 6ms timeframe but occassionally it takes >150ms or even 500ms. This happens about once per second when I am connected to the PC over a Remote Desktop Connection. It is quite rare when I am not connected over RDP (once per 30min). However, I need this loop to always execute in 10ms or less. I also notice the problem more when I am navigating the FP by changing tabs or scrolling through a string indicator.

I am running Labview 2013SP1, 32-bit. The PC has 8 cores and plenty of free RAM and processing power. My program has 2 while loops that contains DAQ functions and they appear to execute fast with no problems; it then has a couple of consumer loops; and then it has 12 processing loops that run in parrallel though 95% of the time only 2 are in use at a given time. I reduced the processing loops to 4 and still saw the same jitter. 

I tried using a Timed Loop structure with 2ms period; I also tried manually changing the thread it runs on; I removed all FGVs and SubVI references from the loop. It does contain one local variable that it reads from (an encoder count) which a DAQ loop writes to about every 5ms. 

Any thoughts on how I could make this loop execute more consistently without changing my hardware platform?

Thanks.

[Blokk]

Use a real time system, not Win OS, this is a short & not too accurate answer.

Some more readings for you:

http://digital.ni.com/public.nsf/allkb/8F35B8099427B48686257A8B003A72D8

edit:

oh, I read your post too fast, sorry 🙂

I find it strange that you experience 100 ms or more difference in this while loop. What do you have there?

edit2:

"It does contain one local variable that it reads from (an encoder count) which a DAQ loop writes to about every 5ms. "

Hmm, it is not 100% clear, so you have a kind of consumer loop, and you would like to time it so it executes exectly at 5 msec speed? And this consumer while loop reads a value via local variable, which value is created by a producer loop with some DAQ functions?

What is your goal here? In similar cases I would let the producer DAQ loop to time the consumer loop: create a time stamp at the producer loop, and send this time stamp + the encoder value to the consumer loop via a Queue. But of course all depends on your actual case / requirements...

Could you give more details? Or even upload a small project, where you are able to reproduce the behaviour?

edit3: but for sure, using local variable should be avoided usually...I feel this is the case in your application too...

[nathand]

In what execution subsystem do your while loops run? At what priority are they running? If they're running in the user interface thread, then it's not surprising that user interaction such as scrolling causes delays. Even if those loops run in a different execution system, if they call a function that requires access to the user interface thread, then they may pause while waiting for it. It also wouldn't surprise me if any user interface actions take longer when connected over remote desktop.

For the least chance of interference, the loops should be in their own subVIs, set to run in an execution system that is not the user interface nor "same as caller," and at high priority. The VI that calls those subVIs should run at a lower priority, and in the user interface thread if it handles user interaction.

Timed loops offer no benefit on Windows, and can actually be slower, because everything in a timed loop executes serially in a single thread, whereas in a normal While loop, the compiler can parallelize operations when possible.

With all that said, Windows is not a deterministic operating system, and you'll never get completely consistent fast loop execution.

[Hooovahh]

Other have already said it, "When you need a real time system, you need a real-time system."  You can't control what Windows does, it may choose to scan your harddisk, defrag, ping the network, or choke and die on a driver at any moment.  You can't rely on it to be consistent.  Generally you can expect a loop rate of less than a couple hundred milliseconds, but if you told me your mouse and keyboard locked up and didn't respond for 10 seconds I wouldn't be surprised.

That being said what does your code look like?  There's a chance that it doesn't need to run at that tight of timing.  If you are needing to respond to another device in that amount of time you have a problem.  But if you are just trying to take DAQ use the hardeware timing to get the speed you need.  Or if data is coming in on a COM port or TCP, rely on the buffer of the hardware.

[mikeporter]

A common mistake people make is to equate real-time with really fast. While many real time systems are really fast, that is not the point.

You need something to happen every by God 10 ms. Not 11 and not 9.5. That may not be very fast as computers go, but that is what real time is for: being able to always do something at a very deterministic rate.

Mike...

[chris_1243]

These are all great responses. Here are some answers:

1. The While Loop, which I will call my "Results Loop", needs to run faster than 20ms at any given time, preferrably, and typically runs at 5ms most of the time. It randomly runs at >150ms and that is the problem I'm having. It happens much more frequent when connected over RDP or when interacting with the UI. So I want the loop to run between 5ms and 20ms at all times. 

2. My while loop currently runs in the same VI as the UI. I would like to get it off of the UI thread--how do I do this? Only timed loops appear to give me the option of specifying a thread and I do not prefer to use Timed Loops if I can help it. Do I need to call the VI using an VI call node and specify it from there?

3. My application works like this: a DAQ loop reads an Encoder count and uses a notifier to trigger a Camera acquisition loop. The Camera Acquisition loop queues an image (note: I realize queueing images is not a great idea but I believe I am handling it appropriately) which goes to one of ten processing loops. My system allows up to 10 processing loops to run in parallel at any given time because some inspections take longer than others and I need to handle the product throughput.. Right now, only 2 processing loops are typically being used though and the others sit idle. The processing loop produces a "pass/fail" result that must then be sent to a DAQ output card as a discrete ouptut to a PLC. The "Pass/fail" result is what is being handled by my problematic loop. I first created a FGV to handle the results from the 10 possible processing loops and then a separate While Loop (my "Results Loop") that handles these results. I recently removed the FGV and now have the processing loops using a queue to send the results to my Results Loop.

4. So what does my Results Loop do? Each result that comes in (on the queue) has an encoder count that was corresponds to the moment the image was taken. I add a rejector distance to this (say, 8000 counts) and track the result until it gets to the rejector. When the product gets close to the rejector, the Results Loop unloads the result from an array which another loop (a DAQ) then handles. I tried several things but currently my Results Loop sits on a "Dequeue Element" that times out in 5ms before it runs through a normal process of checking if any of the results it currrently has are near the rejector and need to be unloaded. If ever the queue has an element, it just takes that element and loads it into an array. I need this loop to respond fast but it does not have to respond exactly the same every time. It can respond 5ms one time and 15ms another time, but it cannot take 150ms+ because by then I can completely miss sending a result. 

5. One of the reasons I have my code set up this way is to use Encoder tracking for the inspection and results. Since inspection time can vary and must run on parallel threads, it is preferrable that my results do not come out overlapping each other when one product takes longer to inspect than another. So I queue up results and spit them out based on encoder counts so that they consistently are spaced apart. 99% of the time this is working great, but I need to get rid of this ocassional 150+ms iteration in my loop and I want to remove it from running on the same thread as the UI.

6. How can I tell which thread number the UI is running on?

[nathand]

---

chris_1243 wrote:

2. My while loop currently runs in the same VI as the UI. I would like to get it off of the UI thread--how do I do this? Only timed loops appear to give me the option of specifying a thread and I do not prefer to use Timed Loops if I can help it. Do I need to call the VI using an VI call node and specify it from there?

...

6. How can I tell which thread number the UI is running on?

---

Are priority, threads, and execution systems not covered by the CLD exam? Oh well. VI priority and and execution system are set in VI properties, under the Execution category. This is set on the VI level, so you should move the high priority loops into their own VIs, set the execution properties, and call them like any other subVIs. You noted previously that you tried to remove all subVIs from your high-speed loops - there's not that much overhead in calling a subVI (unless you were actually calling them by reference, but why would you be doing that?) so that probably didn't help.

Threads aren't assigned numbers in LabVIEW. The UI thread is a specific, special thread. You may find this article helpful: "How Many Threads Does LabVIEW Allocate?"

If it's specifically your "Results Loop" that's causing the problem, can you share the code for that loop specifically?

I would investigate whether there's a way to do this in hardware. For example, if your encoder speed is constant and your DAQ card supports waveforms or timed IO, you might be able to write a digital waveform that turns on the output at the correct time. Even if the encoder speed isn't constant, you could possibly use the encoder as the clock for the digital IO so that it would turn on after a particular number of pulses.

[natasftw]

1) If you want to work in a determinsitic setup, you've received your answer.  Stop using a non-deterministic OS.  You've pretty much explained your problem and it's a simple answer.  The problem appears when you ask Windows to do something more by either working with networking or scrolling through the VI.  Why would you be surprised that an OS that's built on prioritizing response to the user is showing slower behavior in your loop when you're asking it to respond to you?  That's what WIndows is designed to do.  It's acting exactly as designed.  If this isn't what you want, stop using Windows.

2) This answer sounds like you're not understanding LabVIEW's handling of threads.  If you drop two parallel while loops, they're not operating in the same thread.  That creates parallel processing.  Similarly, the loop isn't in your UI thread.  There's a thread dedicated to updating your UI.  You've already got it off your UI thread.  You don't need to do anything else.

3-5) Honestly, this doesn't really help find a fix for the problems you're having.  They're all answered already independent of the application.

6) I'm not sure why this would help you.  Knowing the thread won't change the root cause.  You're focusing on things that won't get you closer to a solution.  What you need to do is make a decision.  You've got two items that you MUST choose one or the other.  Do you want determism or do you want to remain with your current platform?  Windows isn't deterministic. There's no such thing as a WIndows determinism problem.

You also probably want to get a better understanding of Timed Loops.  Using them outside of RT/FPGA is an incorrect usage.  They aren't giving you what you think they are.  The power of the timed loops is priority in RT and single cycle processing in FPGA.  Neither of those apply to your application.  If you ultimately decide to move to RT, as you should, you'll also want to avoid abusing this structure.  Use it for your priority loops and while loops for the remainder.

[natasftw]

---

@nathand wrote:

Are priority, threads, and execution systems not covered by the CLD exam? Oh well. VI priority and and execution system are set in VI properties, under the Execution category. This is set on the VI level, so you should move the high priority loops into their own VIs, set the execution properties, and call them like any other subVIs. You noted previously that you tried to remove all subVIs from your high-speed loops - there's not that much overhead in calling a subVI (unless you were actually calling them by reference, but why would you be doing that?) so that probably didn't help.

---

They're neither covered, nor needed, for the CLD exam.  The CLD focuses primarily on file I/O and timing for functionality and then documentation and style.  The rest is just putting together a state machine.  There's nothing in terms of performance being graded directly.

It's important to note that changing those settings only serves to tell Windows which of these you wish to prioritize.  It doesn't change that Windows is STILL in control of your priority.

Building on the minimal overhead conversation, you can always inline your subVI to get the performance of a single VI with the code readablility of subVIs.

[chris_1243]

As Labview handles the thread selection automatically, I've never had to think about it until now. The Timed Loops ask for a number when specifying a thread so I suppose that is where my confusion came in. I tried increasing the execution priority of a FGV that I had inside my Results Loop but it did not occur to me that I should actually make my result loop a subVI itself and specify it at a higher priority. I will give that a try.

I think you're correct about trying to use a hardware solution. I originally investigated trying to use the Encoder count to produce some sort of event or a clock to time my I/O more precisely, but I could not find any good examples and may not be using the right hardware. I am using the NI PCIe-6341, X-series. After finding little information, I decided to try a software solution instead as I've not had much trouble that way in the past.