Slow Loop Rate / Low CPU Utilization

[wttester]

Hi everyone,

I have a large data acquisition project running on LabVIEW 2019 on a Windows 10 8-core Xeon PC. The data collection and storage functionality is essentially a producer/consumer queue. This system was handed down to me to manage and update.

The Producer gets 2048 data from an external data acquisition system via a DLL, does some processing, and puts statistical data and timestamped raw data into queues. Not sure why there is a queue flush right after enqueueing, but this seems to work. This is all in a 200ms loop.

The Consumer Previews the queue and assembles the data into an array in a 200ms loop. Total duration is 10 sec, so it loops 50x each time we choose to collect data. The queue flush is handled by the Producer. There are many other consumers as well (i.e. displays) but I'm focusing on this one. Again, this is quite a large project.

Everything seems to work fine until I run a relatively process-intensive display. The front panel is shown to convey complexity - it has 2 real-time displays and an FFT running in a 200ms loop. Running this display bumps up all the other loop rates to the 300ms range. I can see that the code can be improved (e.g. separate collection and processing into parallel loops) and maybe that display is too intensive. But then I look at CPU utilization and it is barely cracking 15%. Out of 16 processors, about half is not doing much and only one has high utilization. I've also tried increasing the execution priorities of the producer and consumer and decreasing the priority of the display - this doesn't seem to help.

Any thoughts on what the root issue is here? Is something not optimized or set right? Or am I simply doing too much processing in the display loop? Another thing is I used to run code on LabVIEW 2013 on a Windows 7 PC and I've noticed slower loop speed since switching to LabVIEW 2019 on this windows 10 PC. Any help would be appreciated.

[BertMcMahan]

With only a picture of your code we can only guess at what's slowing things down.

-Are you updating the plots every single loop? Any antiailiasing on the plots? Those can slow things down.

-Are you saying the producer itself slows to 300 ms, or just the other displays?

-Are you using a lot of property nodes to do things?

-Are you using any of the DLL's in your display thread?

-Is that a table control in the display? If you're updating it one element at a time, that'll really slow things down unless you defer panel updates. (Even then, sometimes specific UI elements can be slow, though IIRC that was Trees with LOTS of nodes)

It might be worth reading up on the UI thread and the root loop.

If I had to take a wild guess:

-Open your DLL

-Make sure it's set to "Run in any thread" (otherwise, the DLL runs in the UI thread, which means it can get bogged down if the UI is updating a lot of things)

[GerdW]

Hi wt,

---

@wttester wrote:

The Producer gets 2048 data from an external data acquisition system via a DLL, does some processing, and puts statistical data and timestamped raw data into queues. Not sure why there is a queue flush right after enqueueing, but this seems to work. This is all in a 200ms loop.

---

The whole queue handling is "interesting" (aka bogus)…

• Why do you need to create an array of queues when you need just one queue (PI660_avg_data in producer)?
• Why do you use lossy enqueue when the queue is defined unlimited?
• Why do you flush the queue right after enqueing an item???
• Why do you use the QueuePreview function to read queue elements?
• Why is there a wait function in the consumer loop? In most cases it makes more sense to use the QueueRead function to time the loop, aka process queue items as fast as possible!
• Reading from two queues in a loop (like you do in the consumer) might result in blocking (aka slowing down) the loop…

In general: you will get more/better help when you attach real code instead of just images of parts of code…

[Yamaeda]

---

@wttester wrote:

The Producer gets 2048 data from an external data acquisition system via a DLL, does some processing, and puts statistical data and timestamped raw data into queues. Not sure why there is a queue flush right after enqueueing, but this seems to work. This is all in a 200ms loop.

The Consumer Previews the queue and assembles the data into an array in a 200ms loop. Total duration is 10 sec, so it loops 50x each time we choose to collect data. The queue flush is handled by the Producer. There are many other consumers as well (i.e. displays) but I'm focusing on this one. Again, this is quite a large project.

"but this seems to work" - apparently not. 😉

Flushing the queue after enqueing is wrong. Barock. Silly. Face palm worthy. If there's any possible use for it, it should be in the consumer, after the data has been handled. But as already mentioned by others, it should just be a Dequeue as it'll remove and use an element and it should have no wait, as Dequeue waits as long as it needs to by default.

I would guess it 'worked' earlier by pure luck, the consumer managed to pull the preview before it was flushed, and now it has 50% success, or something similar.

Just use a normal Enqueue and Dequeue and i'll bet a coffee that it works. 🙂

[wttester]

Thank you for your feedback! So it looks like the problem is with the coding of the Queues. And I'm getting quite a roasting over that!

Yeah, many issues with the code. This project was passed down to me. It is huge (hundereds of VIs), is very non-modular, and is in continuous use at our facility. I learned LabVIEW about 2 years ago and have fixed many issues with it. I try not to mess with the code since I've been burned with un-intended consequences. This is a serious problem that I must fix right now, however. I'm planning a completely new system using Real-Time, but that needs to come later.

OK, so what I am hearing is:

1. Queue - yeah, messed up. I will need to do it right. Producer places data in queue, consumer de-queues. However, what is not clear to me is this: If I have multiple consumers of the queue (i.e. other displays), then which consumer does the de-queueing to guarantee all consumers get data before it is dequeued?

2. DLL Thread: yes! they do run on UI thread. Will change to "run in any thread."

3. Parallelism: Need to have separate loops for each queue.

4. Provide code. Please see attached zip file and read "readme.txt." Again, this project is truly huge. I did my best to scale down to just what is needed but include all the relevant VI's. The top-level VIs are PRODUCER#1, PRODUCER#2, CONSUMER, and REAL TIME DISPLAY. Originally, I only disclosed one producer, but there are two and I thought that would be important.

Again, any further feedback is appreciated. Let the roasting commence.

[Mark_Yedinak]

---

@wttester wrote:

Thank you for your feedback! So it looks like the problem is with the coding of the Queues. And I'm getting quite a roasting over that!

Yeah, many issues with the code. This project was passed down to me. It is huge (hundereds of VIs), is very non-modular, and is in continuous use at our facility. I learned LabVIEW about 2 years ago and have fixed many issues with it. I try not to mess with the code since I've been burned with un-intended consequences. This is a serious problem that I must fix right now, however. I'm planning a completely new system using Real-Time, but that needs to come later.

OK, so what I am hearing is:

1. Queue - yeah, messed up. I will need to do it right. Producer places data in queue, consumer de-queues. However, what is not clear to me is this: If I have multiple consumers of the queue (i.e. other displays), then which consumer does the de-queueing to guarantee all consumers get data before it is dequeued?

2. DLL Thread: yes! they do run on UI thread. Will change to "run in any thread."

3. Parallelism: Need to have separate loops for each queue.

4. Provide code. Please see attached zip file and read "readme.txt." Again, this project is truly huge. I did my best to scale down to just what is needed but include all the relevant VI's. The top-level VIs are PRODUCER#1, PRODUCER#2, CONSUMER, and REAL TIME DISPLAY. Originally, I only disclosed one producer, but there are two and I thought that would be important.

Again, any further feedback is appreciated. Let the roasting commence.

---

DO NOT have multiple consumers of a single queue. You may have multiple tasks posting to the queue but you should ALWAYS have one and only one task/process which is dequeueing. (There are some extremely rare but valid cases where you can have multiple consumers but they are the rare exception.) Multuple consumers will be undefined behavior because it is nondeterministic as to which consumer gets the data. Only ONE consumer will ever get the data unless you are using queue preview which is  simply the wrong way to use queues.

If you need multiple consumers you should use a notifier. However with a notifier you cannot buffer the data and your consumers will only get the last value which was posted to the notifier.

[santo_13]

General advice,

• Parallelize as much as possible
  • In state machine, things are pretty sequential by the error wire and you're not taking advantage of dataflow and inherent parallelism
• If content of For Loop takes any longer than few ms per iteration, make them parallelized
• Your original signal analysis part slowing things down
  • If certain analysis takes 300ms, then it requires 300ms, unless you optimize the analysis part

As a day job, I develop Test programs for high-site count Semiconductor Production test and choosing some best practices we could bring down execution time drastically. So, I would advise to start benchmark the execution time of components to identify the long pole in the tent and shorten it.

For example, there are three process running in parallel (A, B, C), if A takes 100ms, B takes 20ms and C takes 300ms, you can say all three completed only at the end of 300ms though they are in parallel. This could be your case.

[Kevin_Price · Accepted solution]

---

1. Queue...     what is not clear to me is this: If I have multiple consumers of the queue (i.e. other displays), then which consumer does the de-queueing to guarantee all consumers get data before it is dequeued?

The answer is that if you have multiple consumers they should NOT be consuming from a single shared queue (see footnote below).  There are better mechanisms for distributing data to multiple consumers.  Here are some:

- 1 dedicated queue per consumer.  Producer enqueues its data into all of them.  Consumers each release their queue when time to exit and all queued processing has completed.

- dynamic events (a.k.a. user events).  1 overall event reference.  Producer generates the event (which will include the data).  Each consumer registers for and handles the event.

- Use 1 single Notifer for the subset of consumers that can withstand lossy data.  In my experience, many live display elements tend to fit this definition.  There's generally a lossless queue for data logging, but multiple display elements can share a single Notifier.

- some variety of global variable.  Producer writes to it.  Consumers read from it in a polling loop because there's no inherent "alerting" mechanism such as you get with Queues, Events, and Notifiers.

-Kevin P

P.S.  Here's the footnote.  Yes, there are *some* exceptions where multiple consumers *do* consume from 1 common queue.  Your situation is not one of them.  If curious, I think the pattern might be called something like Promise / Workers.

[mcduff]

---

@wttester wrote:

Thank you for your feedback! So it looks like the problem is with the coding of the Queues. And I'm getting quite a roasting over that!

2. DLL Thread: yes! they do run on UI thread. Will change to "run in any thread."

---

That will speed things up provided the DLL is thread safe. You should check, if possible, with whoever wrote the DLL. Weird things may happen if it is not thread safe.

mcduff

[drjdpowell]

It looks to me like the original programmer was using Queues to get Notifier-like behaviour.  The way it is done is at best confusing, though, and likely has strange timing effects due to the "erase after writing" thing.  So changing to use actual Notifiers should be an easy improvement that might solve problems.