Enqueued elements are not dequeued in separate loop

[fflo]

Hi, I have two concurrent while loops: the first one is for enqueuing elements from data acquisition. The second one is supposed to run every second (lower speed) to process the enqueued elements which is writing them to a local file.

For some reason there is an unexpected behavior resulting in the accumulation of hundreds of elements in the queue without being dequeued, UNTIL the first DAQ loop is done. After that I can see the number of elements decreasing again.

How can I accomplish fully concurrent enqueue and dequeue of elements?

[paul_cardinale · Accepted solution]

You seem to be asking; ‘If I enqueue elements faster than I dequeue them, why does the queue fill up?’

[BertMcMahan]

We can't help unless you attach your code. Queues will let you dequeue things immediately after you add them, so you likely have something preventing that in your code somewhere.

Attach it here and we'll take a look.

[fflo]

Yeah, my bad, I haven't seen it.🙄

[Mark_Yedinak]

A good option to help mitigate this is rather than dequeue a single data point flush the queue instead. It will give you all of the enqueued items in an array.  Then you just process that array.

Another option is don't gate your processing loop with some arbitrary delay tat impacts the loop execution. Rather, dequeue elements as quickly as they arrive and place them into an array and then process that array at your desired intervals. This is basically a different implementation of the first suggestion as the end result is essentially the same.

[crossrulz]

---

@fflo wrote:

The second one is supposed to run every second (lower speed) to process the enqueued elements which is writing them to a local file.

---

The point of the Producer/Consumer is to be able to process the data as fast as possible when data is available.  A consumer loop should never be time limited other than waiting for data to come in from the producer loop.

[aputman]

You state that the consumer loop only runs after the producer loop is finished.  This would indicate a dependency between the output of the producer loop and the input of the consumer loop.  THINK DATAFLOW.