Process freezes for a few seconds

[Les__Bartel]

Windows 7

LabVIEW 2012 (12.0), runtime engine f3 std

My compiled app freezes every 14 seconds for anywhere between 0.5 to 3 seconds. CPU usage is steady in the single digits (typically less than 5%). My application has 52 queue driven state machines and several windows. All graphics output stops in the LabVIEW windows for this process. Windows Task Manager continues to update. When my application wakes up again, processing continues and there is a brief backlog in some of my queues (expected, since when processing continues, the data that was coming in on a network connection catches up).

What could be causing this? Is the RTE collecting garbage?

 - les

[Mark_Yedinak]

Most likely it is something in the code. However without seeing the code it would be impossible to say. One of our applications is quite complex and uses lots of background tasks with queues as well and we do not experience the same behavior. Is it possible that you have a subVI that is getting called by all of your tasks that is not reentrant? If it is not reentrant and multiple subVIs call it at the same time they will be held off and only one piece of your code will execute. When it is complete another task will run while the others are still blocked.

[CoastalMaineBird]

I have an app with 72 instances of a single queue-driven State-machine VI all simultaneouly running (LV2010), and have no such problems.

You say "compiled app" - does it happen in the Dev Sys?

How regular is the "14 seconds"? (is it 14, 13, 15, 14..., or is it 14, 14, 14, 14...)

How about memory (RAM) - does the task manager say that usage is growing and growing and growing...?

Are you writing to disk?  If so, try to disable that with conditional disables (If DebugDisableDiskWriting == TRUE, then...) and see if anything changes.

Basically you've got to find something that changes the 14 seconds, in order to track it down.

[Les__Bartel]

On my development computer (12 logical cores), the pause occurs on a less frequent and less regular basis (running in the LabVIEW development environment). On 4 samples, the delta between pauses was 44, 43 and 59 seconds. I am not able to attach the hardware under test to my development computer, hence the need to deploy an executable to the test system. The test computer has 4 logical cores. We gather information on one NIC and push some of that data out a different NIC. We have discovered an occaisonal backlog of TCP traffic on our outbound NIC and are investigating the receiving hardware. This backlog shows up in one of the queues, and will catch up when the data rate drops.

The timing of the pauses is very predictable on the run-time system. It runs for 12.5 seconds and pauses for 2 seconds. During the pause, the network stack faithfully buffers the data and feeds it to us when we come back.

I also notice that if I press the mouse button down on the title bar (as if to move the window), my process pauses briefly. I have a clock running on the front panel, reporting time in 10's of milliseconds. This clock will stop for around 1/2 second.

Reproduce this by placing a 'Format Date/Time String' vi in a while loop. The format string should include %2u to report milliseconds. Note, though that this appears to be a completely separate issue from the pauses that are problematic for us.

[CoastalMaineBird]

We have discovered an occaisonal backlog of TCP traffic on our outbound NIC and are investigating the receiving hardware. 

Does a wait at TCP WRITE cause your program to pause?

Find out by turning your TCP WRITE into a sequence: 1... turn boolean indicator ON, 2... TCP WRITE, 3... turn boolean OFF.

Does that light correspond with your unwanted pauses?

If so, then you've found the problem.

[CoastalMaineBird]

The timing of the pauses is very predictable on the run-time system. It runs for 12.5 seconds and pauses for 2 seconds.

Try disabling whatver output your program is producing, temporarily.

Use conditional disabled to turn off TCP WRITE, any DISK WRITE, any GRAPH operations, whatever you're producing.

That will answer the question of whether you aren't getting the data you need, or whether you can't dispose of the results.

[Les__Bartel]

The plot thickens...

The computer has 3 NICs. Everything works fine with one or both instruments on ONE NIC. Unplug the non-essential instrument from the switch and connect it to a 2nd NIC (either one of the remaining NICs), and the network stops briefly every 14 seconds. Data is buffered from the essential instrument and we catch up when the network wakes up again. Note that the non-essential instrument that gets plugged into the other NIC is not configured properly to talk on that NIC. If it is configured to talk on the 2nd NIC, it does communicate properly, but we still see the freeze every 14 seconds on both instruments simultaneously.

This is beginning to look like a Windows network problem.

I did find mention to a CAR (313508) that may be related, but I haven't been able to find much info about that CAR.

[Basil-B]

Have you tried CoastalMain's suggestions in the previous two posts? The boolean should help us narrow down where the problem is arising, whether its from the read processes being slow on the network or whether its the writing and graphing properties. 

[billko]

---

@Les__Bartel wrote:

The plot thickens...

The computer has 3 NICs. Everything works fine with one or both instruments on ONE NIC. Unplug the non-essential instrument from the switch and connect it to a 2nd NIC (either one of the remaining NICs), and the network stops briefly every 14 seconds. Data is buffered from the essential instrument and we catch up when the network wakes up again. Note that the non-essential instrument that gets plugged into the other NIC is not configured properly to talk on that NIC. If it is configured to talk on the 2nd NIC, it does communicate properly, but we still see the freeze every 14 seconds on both instruments simultaneously.

This is beginning to look like a Windows network problem.

I did find mention to a CAR (313508) that may be related, but I haven't been able to find much info about that CAR.

---

Do you think that maybe configuring the "non-essential" instrument to properly work on the NIC should be part of the troubleshooting process?

[Les__Bartel]

---

@billko wrote:

---

@Les__Bartel wrote:

The plot thickens...

The computer has 3 NICs. Everything works fine with one or both instruments on ONE NIC. Unplug the non-essential instrument from the switch and connect it to a 2nd NIC (either one of the remaining NICs), and the network stops briefly every 14 seconds. Data is buffered from the essential instrument and we catch up when the network wakes up again. Note that the non-essential instrument that gets plugged into the other NIC is not configured properly to talk on that NIC. If it is configured to talk on the 2nd NIC, it does communicate properly, but we still see the freeze every 14 seconds on both instruments simultaneously.

This is beginning to look like a Windows network problem.

I did find mention to a CAR (313508) that may be related, but I haven't been able to find much info about that CAR.

---

Do you think that maybe configuring the "non-essential" instrument to properly work on the NIC should be part of the troubleshooting process?

See my bolded text. I tried to say that I ran it both configured and not configured.