I'm working with a producer consumer design to read data off a DAQ and then process in a separate loop (number crunch, update plots, write results to file). The data acquisition rate can be highly variable, so there are some cases where the producer fills the queue faster than the consumer can handle it (one element at a time). So I switched the consumer loop to use flush queue instead of dequeue and batch process all the recent elements. This works well and allows much faster operation. The downside I've noticed is that, when the producer loop is running much slower than the consumer loop, flush queue generates executes with no data to pass.
What I'm wondering is if there is any way to wait for queue elements (as dequeue element does) and then flush the queue, without using too much overhead. I imagine I could use the queue status in a loop and then flush the queue, but I'm guessing this would lead to a significant overhead when I'm running at higher speeds.
The only other solution I can think of is to put the file writing in a third loop with another queue since it is probably what is slowing down the main consumer loop and then put the main consumer loop back to processing one element at a time. But it seems like there should be a simpler way to do this.
Any thoughts, suggestions, or generally better ways to handle this?
Thanks,
Tim
