Flush queue after element = Value

[ATE-EGNE]

Hey All, I have a queue of clusters that each contain an enum and a variant. When an event occurs I want to search the queue for a specific enum+variant  combo and clear the queue after that point. Ideally, I'd like to have minimal impact on dequeueing elements while the process occurs. 

To add additional complexity my consumer loop sometimes enqueues elements in the front of the queue. However I could recode this of it makes a solution impossible

[nclark]

Hey. Could you maybe add a picture and/or the VI? If you add the VI, please save it as 2015 or earlier because I can't open 2016 VIs.

[altenbach]

Can you provide a bit mode detail?

How big is the queue typically? What feeds the queue and what does the data represent?

---

@ATE-EGNE wrote:

...and clear the queue after that point. Ideally, I'd like to have minimal impact on dequeueing elements while the process occurs. 

---

Clearing a queue will have more than "minimal" impact on the ongoing dequeuing operations. 😄

[Alexander_Sobolev]

There is preview element... if you combine it with semaphore, it will allow to remove only "correct" element.

Another options: flush - requeue elements; make monitor process in parallel or in series.

Or make your own queue mechanism with a "monitor function" that has an option to scan "queued" elements .

[Kevin_Price]

I think it's a little bit of a dicey problem.   

First, here's what happens when you approach it from the consumer side:

   On recognition of the event, flush the queue while retaining the returned array of queue contents.  Search the array for the sentinel value and retain the appropriate subset of the array.

   Now here's why it's a little dicey.  In the moments since you flushed the array, the producer may have added things to it.  Presumably, these things are supposed to be acted upon *after* the retained subset of things you just flushed from the queue.  If so, do a Reverse 1D Array on your retained subset, and then auto-index a for loop to keep enqueing them to the front.  Doing it this way makes sure the original contents happen first, in the original order, and that new producer messages queue up behind them.

   If the producer can enqueue to the front, you're kinda hosed.  There's no guarantee what order will result from this process.

Trying this from the producer side could be worse but it depends on the nature of the consumer.  You've told us that the consumer can also act as a self-producer, so anything it self-enqueues between the time your producer flushes and re-enqueues could be acted on immediately, *before* the stuff that had been enqueued first.  Unless the consumer *always* enqueues to the front (not usually a good idea, BTW), you'll be reassembling the queued messages in the wrong order.  You also can't count on re-enqueuing to the front because the consumer may dequeue some of them before you finishing enqueuing them all, and again they'd be processed in the wrong order.

To handle things cleanly, you'd need to mediate access to the queue contents.  A classic old-school construct for that would be an "Action Engine" which held the queue ref internally and never gave it out to its callers.  Then you could code up your search and clear operation as one of the actions, knowing that no other process could break in and have access to the queue until that action completed.  

The newer-school construct would be a class, but unless you're already up the learning curve on LabVIEW classes, I'd suggest trying an Action Engine first.  Most of the actions will be probably be simple straight-through calls to the native Enqueue or Dequeue.  

-Kevin P

[ATE-EGNE]

---

@Kevin_Price wrote:

I think it's a little bit of a dicey problem.   

   If the producer can enqueue to the front, you're kinda hosed.  There's no guarantee what order will result from this process.

Yeah that's the rub. For my specific application I figured out a bandaid problem specific solution however it wouldn't fix the general problem. I'll have to read up on classes more in case I need to do this again. The idea of look in queue -> find element -> perform action after element would be awesome if it could work.

[cbutcher]

Perhaps when the event occurs you can use 'Get Queue Status' to find the number of elements in the queue, then as you dequeue elements you decrement that value whilst checking each dequeued element against your key values. If the element matches, you can then bulk dequeue the remaining number and just ignore them?

The problem here lies in decrementing too many times, due to front-enqueued elements that weren't counted in the number you start with. A solution is to increment your counter if you front-enqueue something that doesn't match the key, and flush the queue if you front-enqueue something which does match the key.

[Kevin_Price]

Before plunging all the way into classes, I'd say again that an Action Engine would be a viable, general purpose solution to this for a single queue.  There are some small-ish downsides of note:

- The Action Engine would mediate access to 1 single queue.  It sounds like that's all you need.  But if you need more you can copy the Action Engine, change the icon to be distinct, and manage 2 queues.  And so on.

- While one process is occupying the Action Engine to do this atomic sequence -- Pull from Queue->Search->Clear->Push to Queue -- any other process that *wants* to enqueue or dequeue will get stuck waiting until the first process is finished.  That'll take some small but non-zero amount of time.  Honestly, it's unlikely to be noticeable under Windows, but it's worth being aware.

- One recurring "issue" with Action Engines is that the various actions often require different inputs and outputs.  And then the callers need to know which inputs and outputs are relevant for the specific action.  It can get confusing.

   One pretty good solution involves making a library where the core of the Action Engine is private (and not directly available to your larger app), but a set of public functions are made available, one for each action, each using only the relevant inputs and outputs.

   As it turns out, this approach is actually a decent-sized step toward a full class implementation.

-Kevin P

[Kevin_Price]

Accidentally wound up stumbling upon this thread today and would like to correct my past-tense self a bit, for posterity's sake.

There's a MUCH BIGGER looming issue if all queue access is wrapped up inside a single monolithic Action Engine.  Namely, the BLOCKING that would happen for a Dequeue operation with any timeout other than 0 msec. 

Wrapping it inside a class can still be workable, provided that the public Dequeue member function is *reentrant*.

There's also a type of Action Engine approach that consists of a private "core" and a public API held together in a lvlib.  Here, the public API function for Dequeue could be reentrant.  It would need to retrieve the native queue ref from the "core" using a *private* function and then call the native Dequeue on it.  No public functions would expose the native queue ref.

-Kevin P

[crossrulz]

---

@Kevin_Price wrote:

There's also a type of Action Engine approach that consists of a private "core" and a public API held together in a lvlib.  Here, the public API function for Dequeue could be reentrant.  It would need to retrieve the native queue ref from the "core" using a *private* function and then call the native Dequeue on it.  No public functions would expose the native queue ref.

---

Let's be honest.  A queue should only have 1 dequeuer: the owner of the queue.  I am a fan of the private Action Engine with wrapper "command" VIs so other libraries can send commands/data to my consumer.  So my actors end up looking something like this: