One part (thread) of a run-time application freezes for hours until human interaction with PC

[mirola]

Dear community,

We had in our company and at customers’ site the following problem in three cases.

-       Big run-time application (10 MB) made with Labview 2011.

-       Running on Windows XP (in one case on Win 7, but not sure that this case is equal).

-       Same software in all cases, but different hardware (biolab machines).

-       Some PCs are connected to the internet (automatic Windows update should be deactivated), some is not.

Problem:

The main loop of the program freezes for several hours (up to 10) in overnight processes, while at least one other independent loop keeps running. This is documented by some log-files, generated by the program itself and by the firmware of some of the hardware.

The program hangs (in one best documented case) in between two commands to the hardware (DLL -> USB-to-CAN), but a normal communication failure would lead to a time-out error. Besides, the other loop that kept working continued the communication with the same hardware (DLL) normally over the entire time.

Observations:

-       The frozen loop seems to be waken up by human interaction with the PC (mouse movement).

-       The program continues normally afterwards.

-       The problem appeard after the update from an older version of Labview.

Ideas:

-       Could it be that in multi-threading of the program some CPU core hangs while another one continues processing its thread(s)?

-       Nearly the only significant Labview command at the point where the freezing happens is “Waite (ms)” with 0 ms input in this case. I studied posts about that function and its CPU relevance, but didn’t see any clear hint.

Since the problem appeared very occasionally it is hard to investigate. I hope for some ideas from NI or the community.

Best regards.

[mikeporter]

Has this program always done this, or is this a new situation? Any chance of seeing any of the code?

Mike...

[mirola]

Hi Mike,

Thanks for your interest.

As I said, the problem appeared after changing to 2011 (from 7.1, big jump, I know). The code was nearly not changed in that part. One thing that is new at that point of the code is a conditional disable structure, which is empty (default) in this case. A boolean goes in and out, without being connected inside (unwire default output). It is followed by the mentioned 0 ms wait.

I don't think it makes much sense to send pieces of code.

This is a quite nasty issue for us. Customers will loose time, material and money.

mirola

[mikeporter]

What happens if you change the 0ms wait to 1ms? It's really hard to give specific advice without seeing the code.

Mike...

[mirola]

I know that it isn't a simple question. I was hoping that it would ring some bell, e.g. looking at it more from a general Labview-application-under-Windows-on-a-multicore-PC perspective than analysing the code.

What is going on if moving the mouse really unblocks the frozen part of the program?

(Of course, there is no mouse event handler etc. in that part of thecode.)

I could change the 0 ms to 1 ms. But the problem is that it happened very few times and several hours afte starting the program/process.

Therefore its dificult to test and to see if such an modification has some effect. And I was hoping to get some hint.

[mikeporter]

I hear you, does the computer you are running this code on have a screen saver? I have seen this cause problems. Is there anything going on on the network that it might be correlated to? Automated Backup? Virus Scans? What abouth the power saving settings? Are they putting anything to sleep after a couple hours?

This is a long shot, but if something is causing the UI thread in LV to hang, the solution might be to move the code out of that thread. Now anything that interacts with a front panel has to run in the UI thread, so you will need to redesign your code such that the acquisition logic occurs in a separate VI that executes in a different thread. This separate loop would then pass the data it acquires to the rest of the program via a queue or user-defined event.

Mike...

[mirola]

Screen saver was not on, maybe monitor power save after 20 min or so. But it the freezing happened (at least in two cases) hours after the user left the room.

If power saving things are set, they didn't cause any problem for a long time (before the upgrade to rte 2011).

I didn't see any other automatic tasks. There is nothing in the Windows Event Log.

No virus scanner (at least in one of the cases).

I don't know if I understand all what you wrote. The freezing happened (at least in the one best known case) deep down in a sub-sub-sub-VI, not in the (any) front panel VI. The other loop that was running normally during those hours is a part of the main UI front panel VI.

[mikeporter]

The thing is that by default, a subVI executes in the same thread that its caller does, so if the top-level VI is in the UI thread, its subVIs will be as well. When it hangs in the sub-sub-(etc)-subVI, is it always the same one?

Mike...

[mirola]

But can some independent loops in the top-level VI be executed in different threads? So that what you called the "caller" would be a loop of the top-level VI.

If this is not possible, then we are looking in a wrong direction. Because, as I said, another loop kept running.

How would I create "a separate VI that executes in a different thread" if at the end all VI are sub-...-VIs of one top-level VI? In my case, the VI that I called top-level, the one with the UI panel, is actually one level below the overall top-level VI (which keeps its small front panel open, non-closable until the UI panel is closed).

Thank you for your effort so far. I will leave the office soon and will probably not reply again today (CET).

[mikeporter]

Unfortunately, I am flat out of ideas. Unless it always hangs in the same subVI and you can find something there, I don't knwo where else in general to look.

Mike..