LabVIEW
Linux RT: What does "Thread consumption suspected" mean?
Solved!
Hello,
My RT error logs show things like this:
LabVIEW RT Error Report generated 5/05/2016 6:12:14 AM Target code: cRIO-9030 Firmware version: 2.5.0f1 ******************** LabVIEW Error Log ****************** #### #Date: Wed, May 04, 2016 11:05:17 AM #OSName: Linux #OSVers: 3.10.31-ltsi-rt29-2.10.0f0 #OSBuild: 199199 #AppName: lvrt #Version: 14.0.1 #AppKind: AppLib #AppModDate: InitExecSystem() call to GetCurrProcessNumProcessors() reports: 2 processors InitExecSystem() call to GetNumProcessors() reports: 2 processors InitExecSystem() will use: 2 processors starting LV_ESys2_Thr0 , capacity: 24 at [3545175933.37509394, (11:05:33.375094000 2016:05:04)] starting LV_ESys2_Thr1 , capacity: 24 at [3545175933.37509394, (11:05:33.375094000 2016:05:04)] starting LV_ESys2_Thr2 , capacity: 24 at [3545175933.37509394, (11:05:33.375094000 2016:05:04)] starting LV_ESys2_Thr3 , capacity: 24 at [3545175933.37509394, (11:05:33.375094000 2016:05:04)] starting LV_ESys1248001a_Thr0 , capacity: 1 at [3545175945.01412582, (11:05:45.014126000 2016:05:04)] Thread consumption suspected: 2 Try starting 1 threads starting LV_ESys2_Thr4 , capacity: 24 at [3545175980.54276419, (11:06:20.542764000 2016:05:04)] Thread consumption suspected: 3 Try starting 1 threads starting LV_ESys2_Thr5 , capacity: 24 at [3545175984.56444788, (11:06:24.564448000 2016:05:04)] Thread consumption suspected: 4 Try starting 1 threads starting LV_ESys2_Thr6 , capacity: 24 at [3545176207.70616293, (11:10:07.706163000 2016:05:04)] Thread consumption suspected: 2 Try starting 1 threads starting LV_ESys2_Thr7 , capacity: 24 at [3545176256.36498308, (11:10:56.364983000 2016:05:04)] Thread consumption suspected: 7 Try starting 4 threads starting LV_ESys2_Thr8 , capacity: 24 at [3545184996.81207800, (13:36:36.812078000 2016:05:04)] starting LV_ESys2_Thr9 , capacity: 24 at [3545184996.81207800, (13:36:36.812078000 2016:05:04)] starting LV_ESys2_Thr10 , capacity: 24 at [3545184996.81207800, (13:36:36.812078000 2016:05:04)] starting LV_ESys2_Thr11 , capacity: 24 at [3545184996.81207800, (13:36:36.812078000 2016:05:04)] Thread consumption suspected: 12 Try starting 2 threads starting LV_ESys2_Thr12 , capacity: 24 at [3545193551.72287989, (15:59:11.722880000 2016:05:04)] starting LV_ESys2_Thr13 , capacity: 24 at [3545193551.72287989, (15:59:11.722880000 2016:05:04)]
What does "Thread consumption suspected" mean? Is it something I should be trying to eliminate?
Thanks for the bump Mark, and thanks for moving the discussion along natasftw.
@natasftw wrote:It sounds like the thread is consumed and needs to wait to process whatever else you're trying to run.
Well, my question was basically this: What does it mean to "consume" a thread? I haven't seen this terminology before, and I Googling hasn't gotten me very far (here's a sample search query). I've heard of CPU time being "consumed", but not of threads being "consumed".
As I understand it, a process can have many more threads than CPU cores without issues; the operating system will allocate CPU time to each thread as it sees fit. Furthermore, threads can be created and destroyed on demand, or long-running threads can be reused to run different tasks to avoid the overhead of creating/destroying threads. Where does "consumption" sit within this model?
@natasftw wrote:It sounds like the thread is consumed and needs to wait to process whatever else you're trying to run.
Are you pushing the limits of your processor? Do you see any loops starving?
For this particular application, the Distributed System Manager reports a CPU usage of ~10% when running from source, and ~1% (yes, that low!) when built + deployed. Memory use is low too, at about 35%. There is only one timed loop, with a period of 50 ms. There is one untimed loop with a period of 50 ms. All other permanent loops (about 10 of them) are untimed, and have a period of between 250 ms to 1000 ms (most are 1000 ms). One of these slow loops use Wait Until Next ms Multiple, all others simply use Wait (ms).
Do you see any potential issues with this architecture?
@natasftw wrote:Do you see any loops starving?
I have seen one incident where the untimed 50 ms thread seems to have stopped running while all others were still running... But I'm not able to reproduce that anymore 
We have a definite answer from NI staff! https://decibel.ni.com/content/thread/46426
"That diagnostic indicates that all the threads in the given execution system have been blocked for "a while" (many milliseconds). Prior to LV 2015, that would mean the execution system couldn't do anything else until one of them unblocked. In 2015, we'll allocate an extra thread or two to see if that un-sticks things. It's not necessarily wrong and can be normal, but it's a bit of a bad sign, because if whatever is consuming all those threads keeps consuming the extras too, eventually the execution system will stop making more threads and will block, which could lead to an unresponsive application."
