queue efficiency

[billko]

---

@JKSH wrote:

Hi,

If it's not too hard, you could set up both scenarios and find out which one works better.

Anyway,

• Have you profiled your code to find where the bottleneck lies? Is it in the processing, or the disk-writing?
• What is the "production rate" of your producers?

Note: If you go down the 2-consumers path, make sure your consumers use reentrant VIs only. Otherwise, your 2 loops will block each other.

---

Haha - I did this once when I thought I would be fancy and turn the wait into a subVI so I could have error in and out.  I couldn't understand why my code was running so slowly until I noticed the "timeout period" was roughly equivalent to my longest wait.  Once I figured out the time period, it took me two seconds to figure out that the wait VI needed to be re-entrant.  Even though the wait itself doesn't care if it's called in multiple places at the same time, my subVI sure did.  😉

[RavensFan]

Bill,

The image in your signature right before your CLD logo is just showing up as a Red X in IE9.  In firefox, nothing is there.

The image is

<img alt=&quot;&quot; src=&quot;http://forums.ni.com/t5/image/serverpage/image-id/140030iAF26F8B6D54CE5E8/image-size/large/is-modera... style=&quot;width: 250px; height: 68px;&quot; />

[billko]

I have a hard time with that because it shows up right in my browser.  I tried looking at it logged out and it showed up okay, too.  I was never good at html so I just used an online editor and copied the resulting source code.  I'll just disable it for now until I can figure out why I can see it but no one else can.

[billko]

---

@RavensFan wrote:

Bill,

The image in your signature right before your CLD logo is just showing up as a Red X in IE9.  In firefox, nothing is there.

The image is

<img alt=&quot;&quot; src=&quot;http://forums.ni.com/t5/image/serverpage/image-id/140030iAF26F8B6D54CE5E8/image-size/large/is-modera... style=&quot;width: 250px; height: 68px;&quot; />

---

I'll just disable it until I can figure out why I can see it but no one else can.  (I even tried logging out in case it was a permission thing.)  I'm not good at hand-coding html so I used an online editor to generate the source code for me.

[RavensFan]

It's possible that the HTML I posted wasn't for the element that was causing the red X box.

I could see the CLD logo okay.  Just that something to the left of it wasn't showing up.  I could have taken a screenshot of it for you, but I didn't think of it at the time.

[GerdW]

Hi Billko and RavensFan,

the CLD logo itself was visible, but in your (Billko) signature was a second image reference just before that CLD logo.

So it would be sufficient just to delete that second (or better wording: first image reference) and keep your CLD logo…

[Hornless.Rhino]

---

@billko wrote:

 That is assuming that the two consumer loops are identical and you don't care which dequeue gets what element.

Any gurus want to help out?  I'm very curious to hear what everyone has to say...

---

Both producers are identical, both consumers would be identical. It wouldn't matter which dequeued where.

---

@GerdW wrote:

Hi Rhino,

dequeuing a queue in two places is only an option when you don't rely on the queue element order. When this point isn't critical in your application you can dequeue as often as you want to…

---

In this case each queue element is independant enough dequeueing in different places won't affect things.

---

@JKSH wrote:

Hi,

If it's not too hard, you could set up both scenarios and find out which one works better.

Anyway,

• Have you profiled your code to find where the bottleneck lies? Is it in the processing, or the disk-writing?
• What is the "production rate" of your producers?

Note: If you go down the 2-consumers path, make sure your consumers use reentrant VIs only. Otherwise, your 2 loops will block each other.

---

I'll do a proper profiling when I get in to work on Monday. The production rate at present is be between 15 and 30 queue items a second. When it's the smaller elements it will handle it fine (and have pretty much no queue build up at all). The more complex elements drop the production rate down a lot, but take longer to process and cause the queue to start increasing.

---

@GerdW wrote:

Hi Rhino,

what about a 2nd consumer, which only handles the "hard and heavy" data packets? Your 1st consumer will hand over those packet to your 2nd consumer by it's own queue…

---

I'll have a look at that. Good idea.

---

@Michael_78 wrote:
Thos sounds like a worker pool. The consumers are clones listening to the same queue. You can launch more clones as the load increases, you can watch the queue size to monitor load. On a multi core or multi pc this is a very scaleable architecture.

---

I'll look in to this.

---

@Thoric wrote:

 To have multiple slaves (consumers) processing from one queue is perfectly acceptable, so long as the order of consumption is not important.

---

There are four (at present) types that at present can be put on the queue (but no more than two at a time). The only real requirement is that items of the same type are dequeued in the order they are put on. So if the enqueue order was  1a 1b 2a 1c 2b 2c 1d 2d, it wouldn't matter which consumer dequeded as long as within 1 and 2 it kept the abcd order.

---

@crossrulz wrote:

My first suggestion would be to make another loop (and corresponding queue) just for writing to disk.  Since that tends to be a "slow" process anyways, it should help releive your consumer loop at least some.  But, again, I do not know what your "complex" commands require of you.

---

That's a good idea actually. If I dequeue and process, and then bump it into another queue it could help a lot. It's more complex in the sense that it requires slightly more processing. I'll do the profiling as suggested earlier, and look at your pure logging queue. I have a feeling that it's the logging of the complex items that's doing it. (Off the top of my head I think there's about a factor of 15 difference in data size to be logged between the simple and complex ones.

---

---

Thanks for your help/suggestions everyone. I'll report back when I have it purring like a well maintained car.

[Shazlan]

A side note... Why not use User Events instead? It is also based on Queues. The best advantage (at least in my opinion) of user events vs. primitive queue is that I can implement 1:N, N:1 and N:N communication method with user events and not limited by 1:1 in Queue.

[Hornless.Rhino]

Ok, just to keep you up to date. I shifted the data logging to a secondary queue and it's now cooking with the environmental energy source of your choice.

Producer===first queue==>Consumer/processer====>second queue===>logger

[billko]

---

@Hornless.Rhino wrote:

Ok, just to keep you up to date. I shifted the data logging to a secondary queue and it's now cooking with the environmental energy source of your choice.

Producer===first queue==>Consumer/processer====>second queue===>logger

---

I like that solution.  Usually the logging is the slowest part, so if you put it in its own queue, it can be the slowpoke and not affect anything else.  Of course, it still has to be generally faster than the other loops - i.e., it can absorb great gouts of information as long as it can be given time to digest it before the next data blitz, or eventually you run out of memory.  There are several different ways of handling this, depending on the level of data integrity you need.