LabVIEW Timed Loop falls waaay behind, unknown delay or error

[LabVU_Dog]

I'm using a Timed Loop in LabVIEW 8.0.1 in Win XP. I have a rather simple subVI that controls a single channel PWM output of a single SCXI relay channel. Period (0.8sec < P < 6 sec) and duty cycle are communicated to the loop, and it turns the output relay ON and OFF with the appropriate timing. The PWM action seems to work fine for controlling my process. This subVI is reentrant, and I dynamically load a number of instances (8 or more) of it at runtime.

Once in a while (every few hours or less) the timed loop seems to "lock up" for ~5200 msec. I found this when the temperature I was controlling would spike. The SW lock-up would have to happen at the same place in code (during the ON phase) to cause this problem repeatedly. I cannot determine if similar events happen in the OFF phase because because there would not be enough temperature response (for ~ 5sec delay) to distinguish from normal process variation. (Normal DC at steady state is <7%)

I put timers and set traps in my SW to troubleshoot this. The DAQ calls are reentrant, and even at the time of the delay, take no longer than 50msec. I have determined that the start time that I provide to LV for the start of the 2nd frame (I use a 2 frame timed loop) is computed correctly.  I cannot figure out what could be causing this delay. I thought perhaps the OS is busy doing it's thing (reporting to Bill or whatever), but nothing else is running on the computer, other than a clean (whatever that means!!) Win XP install. It's also on a LAN that is isolated from the internet.

The delay does not create a LV error that stops the loop! After the delay, the loop picks up where it left off and continues working.

I rebuilt the code using a regular while loop using my own timing. I ran it on the same computer attached to the same HW/process as the timed loop that had the failures. It worked flawlessly. I would have left the issue, except for the fact that a coworker has recently started experiencing the same issue on a completely different task/computer. He's using LV 8.2 on Win XP, and his LV code is compiled into EXEs.

Attached is the reentrant subVI if you care to review. It should be straightforward in terms of operation, and you can see the remnants of debug/troubleshooting code. I'm continuing to troubleshoot.

Please let me know if you have any ideas. Thanks.

[Steven_C]

Hi LabVU_Dog,

I just wanted to clarify clarify a couple of things in an attempt to narrow down the issue.

- On your co-workers computer, is he running the same code that works on your computer (in other words the one that has been re-created using normal while loops)? Is it completely different code but simply using the same techniques that you had used originally (I am asking this because you mentioned that you had LV 8.0.1 and he has 8.2 which may imply that he is doing some development of his own)? Is it simply the 8.2 run-time engine?
- Does he only experience this behaviour when running the .exe files or when he is in the development mode?
- Do his .exes work on your computer?
- Do your .exes run on your computer (or have you only run the program in development mode)?

Please let me know. Thanks and have a great day!

Best regards,

Steven C

[LabVU_Dog]

- On your co-workers computer, is he running the same code that works on your computer (in other words the one that has been re-created using normal while loops)? Is it completely different code but simply using the same techniques that you had used originally (I am asking this because you mentioned that you had LV 8.0.1 and he has 8.2 which may imply that he is doing some development of his own)? Is it simply the 8.2 run-time engine?
****My coworker is developing a completely different application, and using LV8.2. Mine runs entirely on PC. His runs on Compact Fieldpoint, with the user interface on the PC host. He's experiencing delays like I am on the PC host. He uses generally similar techniques, but the code in the timed loops is really pretty simple. Mine has single channel (DAQmx) digital output, his has network comm. When he replaced his timed loop with a regular while loop, the problem disappeared completely, just like mine.

- Does he only experience this behaviour when running the .exe files or when he is in the development mode?
***I don't recall, I'll have to check with him.
- Do his .exes work on your computer?
***We haven't tried, and won't be able to for another week or two as the cFieldPoint HW is in use for an experiment.

- Do your .exes run on your computer (or have you only run the program in development mode)?
***I have only run development mode.

One thing that I did not mention in OP is that when I modified the code with the original timed loop, I started getting strange block diagram artifacts, and when I went to save it, LV told me there were insane objects. So I deleted the loops all together (but left the front panel controls) and started the subVI again. I built it with regular while loop, and it didn't have any insane objects, and worked fine. Because of this, I thought there might have been some corruption in my original subVI. So I rebuilt it with the timed loop and tested it overnight. It still has the same problem.

[Ben]

Hi LabVU_Dog,

I looked at your code and it looked fine at first glance.

Where you are using queues I would use a LV2 set for subroutine priority and use the "skip if busy".

That aside;

I can not say that I have run a timed loop for hours but I have run two timed loops using hardware timing at 1000Hz with no lates after 20 minutes.

If the sub-routine thing does not fix it, I suspect you are dealing with the limitations of Windoze.

On that note, if you watched Performance Monitor while the hit is happening (yeah right he says) you may be able to nail the culprit.

Here is a short list of other things to try

1) Set Windows to optimize background services.

2) Stop Indexing services.

3) Set LabVIEW priority in Task manager.

4) Shutdown Virus Checking.

5) Disconnect Ethernet cable.

6) Set the VI properties of the sub-VI that will run the Timed Loop to "Priority = Time Critical" and "Execution Thread = Data Acquisition".

If any of the above helps, PLEASE report back so we can keep track of what the issue was.

Trying to help,

Ben

PS: Nice code! I am not used to seeing code that good from a non-regular on this forum.

[LabVU_Dog]

Where you are using queues I would use a LV2 set for subroutine priority and use the "skip if busy".

I like the "skip if busy" feature, but I"m not sure how to implement this in my architecture. It would work fine if there were only one instance of the subVI. But, I load many instances. In that case, each instance would have to have some way of knowing which it is, and passing that to the subroutine to extract the info pertinent to it. And the calling code that puts the period/DC into the LV2 global would have to keep track of the same info.

I suspected that this (Check Queue subVI) might be the problem, and recorded the amount of time the "Check Queue" code took to execute. It was never >0msec, even on the occassions that the delay occurred.

PS: Nice code! I am not used to seeing code that good from a non-regular on this forum.

I wasn't that proud of what it looked like when I posted; I had a lot of dead ends from various troubleshooting techniques. When I rebuilt it, I cleaned it up. See attached. Even the new code has the same delay issue.

I'll see about trying the other fixes you describe.

[LabVU_Dog]

I should also note that I tried setting the VI priority to time critical, and Data Acquisition execution system. This did not help. You probably already know that timed loops run at their own priority between Time Critical and High priority, regardless of the subVI you put them in. I also went and configured LV to open as many threads (8 currently) in the combination of priority/execution system as VI instances I'm using in case that was a problem. Further, I set the timed loop priority of VI instances not in use to 1, and others that are in use >1. None of this seems to help.

I observed on several occasions that 2 delays happened within 1-2 minutes of eachother.

[Ben]

To help ID the source of the "hang-up", could you toss the queue operations long enough to satify yourself that they either are a factor in the hang-up or not?

Ben

[LabVU_Dog]

I removed all queue functions from the PWM subVI (and all of it's subVIs) and ran it. I had the same problem. I returned to my original code with the queue functions and disconnected the ethernet cable. I ran it again and had the same problem. I'll keep troubleshooting.

[Ben]

Did you shutdown the indexing of your disks? That is notorious for wasting CPU just so you can find files faster.

Ben

[LabVU_Dog]

Do I do this by unchecking the box on the Properties Dialog for the hard drive(s)? Is there anything else I need to do to disable fast indexing?