Large Application, LabView seems to use only a single cpu core causing performance issues and process delays

[MarvSchl]

Hi All,

I'm hoping for some helpful insights regarding LabView CPU usage and process delays:

What I have:

a large application that runs several process loops (>25) in parallel using a Queued Message Handler (QMH) framework. One of the main functions of the program is to read, write, scale and display channel data from one rt-target (~550 channels). Several channels are displayed in a total of 12 charts. The application receives data at a given frequency from the rt-target (100Hz) via Network Stream.

Issues:

The program functionally works, but it seems, that I'm running into performances issues, causing unexplained delays (up to 30sec) between enqueuing and dequeuing elements of certain queues. One example: A user event is triggered and a command (Command= Cluster with string, I32 and variant) is enqueued at a specific consumer queue, however dequeuing the element at the specific consumer takes up to 0-30s. Additionally, the consumer is in idle mode (does nothing other then "wait for element" with timeout = -1). This pattern of delay did not show for consumers receiving queue elements continuously. I have checked the CPU usage and it seems that LabVIEW uses only a singe cpu core for the application instead of the available 8 cpu cores (one core is at ~90% CPU load, while 7 cores running at ~10% CPU load and LabVIEW uses ~15-20% CPU load in total).

Investigations done:

- I double checked if all vis are re-entrant (shared or pre-allocated clone)

- I have tested the application on three different machines (same system configurations, different number of cpu cores) with same results.

- the delay between enqueue and dequeue is reproducible for all queues, which do not continuously receiving data

- When running a minimal example with multiple while loops or consumers, Labview uses all cpu cores

- I have replaced some consumer loops with timed-loops and assigned them different cpu cores to no effect

- set timeout to 0ms for enqueue element

Questions:

- It seems that LabView uses only one cpu core for the application, but I don't know why?

- Any ideas?

System:

- LabVIEW 2019

- Windows 10

Application:

- QHM framework

- Class usage (LVOOP)

- using subpanels and multiple charts

- charts are updated (100Hz)

Since it is a very complex problem I tried to list only the most important bits and maybe with specific question from your site, we can find the source of the issue and a solution.

Thank you for your help and input!

Kind regrads,

[GerdW]

Hi MarvSchl,

---

@MarvSchl wrote:

What I have:

The application receives data at a given frequency from the rt-target (100Hz) via Network Stream.

Issues:

I have checked the CPU usage and it seems that LabVIEW uses only a singe cpu core for the application instead of the available 8 cpu cores (one core is at ~90% CPU load, while 7 cores running at ~10% CPU load and LabVIEW uses ~15-20% CPU load in total).

Investigations done:

- I have replaced some consumer loops with timed-loops and assigned them different cpu cores to no effect

Application:

- charts are updated (100Hz)

---

What happens when you update your charts with less than 10Hz?

Is there a need to update them faster than you can actually recognize the updates? (The human eye/brain can only recognize upto ~20Hz updates.)

Btw. using TimedLoops on Windows most often makes the problem worse than it helps.

[pallen]

You may have already checked this. 

Have a look at your BIOS settings to make sure Hyperthreading is properly enabled.  

Once had a project where the customer upgraded the PC running the application, only to find that it performed worse than the old one.  

Check anti-virus or other security settings as well.  Maybe it's just me, but I've been finding a lot of performance issues with Windows 10 recently.  

Check the disk usage when you're looking at CPU stats.  Had a few PCs with 100% disk usage problems recently. 

[JÞB]

Tools>>Find parallizable loops.

Just a magic 8-Ball guess.

[MarvSchl]

Thank you for your prompt response. I have to correct, the data rate is 10Hz not 100Hz (big difference, sorry for that). I will update the starting post.

I have tried to reduce the update rate of the charts with different results. First I have to mention that each chart is running in its own consumer loop and is updated with every queue message containing data (currently ~10 messages per second). To reduce the update rate of the charts, I have two possibilities:

1. Parse only every x sample into the chart. I have tried this by writing only one of 10 samples into the chart, so that the chart is updated with 1Hz. In this case, the consumer still receives ~10 messages per second, but the problem still exist. (synchronous display for all controls is disabled).

2. Parse only every x sample into the queue of the consumer containing the charts. I have tried this by writing only two of 10 samples to all loops processing the data (especially the chart consumers). This reduces the cpu load of the cpu core, previously running at 90%. The application runs fine depending on machines single core performance and update rate, but doesn't fix the problem on a weak machine.

I'm running the application on a virtual machine. I noticed, that it doesn't matter how many cores I assigned to the virtual machine, the overall performance and the delay is the same.

[GerdW]

Hi Marv,

3rd option:

Send all samples to your consumer loops. Collect the samples inside the loop in an array. Send that array only once (or twice) per seconds to your chart! This way all samples get transferred and displayed, but with a reduced update rate of the charts…

[JÞB]

Ok, I recalibrated my 8-Ball. 

Focusing in on the single core burning issue.  I would bet the UI Thread is chewing up that core!  Especially when you mention that you have charts updating in subpanel containers.  

TRY a judicious defer FP Updates here and there:) (can't see the whole picture,  use your experience to place the invoke nodes)

[richjoh]

Stuff a lot of "Get Queue Status" in your code to see what's going on with the queues being buffered.  Queues can be difficult to debug since nothing happens and it appears to hang. 

You mention your minimal example works... OK keep increasing from there when does unexpected delays occur, at 10, 20, >25 loops?

[altenbach]

---

@JÞB wrote:

Focusing in on the single core burning issue.  I would bet the UI Thread is chewing up that core! 

---

I was just about to say the same thing. What is the chart history length of all these charts? (550(!) channels in 12(!) charts at 100Hz(!)). The UI thread is single threaded and you are really hammering it. Everything else takes a backseat.

I don't think anyone can give more specific advice unless you show us some code.

[JÞB]

---

@altenbach wrote:

---

@JÞB wrote:

Focusing in on the single core burning issue.  I would bet the UI Thread is chewing up that core! 

---

I was just about to say the same thing. What is the chart history length of all these charts? (550(!) channels in 12(!) charts at 100Hz(!)). The UI thread is single threaded and you are really hammering it. Everything else takes a backseat.

I don't think anyone can give more specific advice unless you show us some code.

---

Charts in subpanels (or tabs) or other containers like splitters, chew the UI thread for double bonus updates.  There is something remarkable about using containers on a GUI that isn't frontmost or Modal.  The UI really does predictably think of the USER first! Perhaps logging or something else really is more important?  You need to especially code around the absent user.

SO, go back , get the "User's Story " and redesign the Front Panel.  HINT: No user wants to process one point twentyone jiggabits per second unless they have a flux capacitor and a Delorean.   (Updated to a multicore Intel 64bit processor)

Sea-story time!  Way back when I was washing dishes for a living *1 I came across a project that had 14 tabs full of individual parameters and graphs.  The user wanted "a big green round color box and an anticipated time for experiment completion easily observed from the hallway through the glass window."

*1 ACTUALLY I was automating a few dish machines for ECOLAB. See them in action on the R&D tour in Mn.