I am hoping I could get some feedback on a project I'm working on. In short, we are using a DAQ assistant to obtain. The thing with our project is that it is acquiring a lot of data. The attached image is what we are currently doing at the moment. Our end goal is to take the numbers from the DAQ, sample compress it so that we don't have too much data, then average those numbers every X minutes. We just found out that possibly using the producer/consumer loops can be beneficial for our large data. The thing is, I'm just not sure how to go about it. We have lots of components in our coding that needs to be taken into consideration, would those all be going into the producer loop as well? Since the only thing we would want in the consumer loop is from the sample compression to the write to measurement file?
The thing with a Producer/Consumer is that you can create stages. Generally, you just want your first loop to acquire the data and dump it into a queue. The second loop should dequeue that data and process it. The second loop can then enqueue its data into another queue so that a third loop can record the data.