I have been trying to use a producer-consumer architecture to read, log, and display data. We are getting 8 channels (more in the future) and 50KHz each channel of data. I have a producer loop reading 1000 samples each iteration and store them in a queue (and a notifier). I want to display all 8 channels of data on 8 waveform graphs to monitor the signals, but when I tried to plot the data in a consumer loop via either queue or notifier, it slows down the acquisition loop and cause the "application is not able to keep up with the hardware acquisition" error. One or two waveform graphs may be okay, but as the number increase they slow down the acquisition till a point of error. This seems to be contrary to the advantage of this architecture, but I don't know what I did wrong. I have attached some images of the program. Any suggestions? Thank you very much!
Inside producer loop:
I've tried using queue and notifier but both gave similar results.
Solved! Go to Solution.
We don't work well with truncated pictures. Can you attach a simplified VI instead?
I doubt it's the core problem, but get rid of the value properties. Completely unnecessary.
Where do you read the queue? Where is this running? What's the hardware? Can you really see 8k of points updated 50x/second?
Did I understand correctly that when the "RT Monitoring" is false, everything runs okay? When you set "RT Monitoring" to true, things slow down?
1. There's a rule of thumb I tend to follow that the data acq read loop likes to iterate at about 10 Hz. It's a pretty good tradeoff to read in reasonably large chunks while also updating displays and indicators at a rate faster than a user can interpret them. (Not a "flicker free" ~30 Hz experience, but plenty fast enough for interpretation and reaction).
So one thing to try is reading 5000 or more samples per iteration.
2. The only thing I notice that might slow down your acquisition loop is the Enqueue. You didn't wire a timeout, and the default is infinite. Did you also set the queue to a fixed size in your call to Obtain Queue? If so, then when the consumer bogs down, the queue will fill up and block the producer loop. (If your queue does not have a size limit, the same kind of issue can *eventually* develop due to the need to use the memory manager to acquire more memory.)
One "trick" I've heard others mention is to pre-fill the queue with a very large # of elements and then flush them before entering your producer loop. This puts the memory manager overhead outside your normal run time. You might also designate a very large fixed size which likely invokes the memory manager at "Obtain" time.
Finally, you could consider a non-infinite timeout on the enqueue. That'd make you miss some data, but it could prevent the fatal data acq error that makes you miss *all* future data.
3. You could increase your acquisition buffer size too, but that's mainly likely to delay the problem rather than solve it. The default buffer size for a 50 kHz acquisition is 100k samples (per channel). That's 2 seconds worth. You might try bumping that up to 10 or more seconds worth, though I suspect you'll only delay the onset of the same error.
4. Using a Notifier for the display consumer is the right approach because Notifiers are inherently lossy and can't cause the queue blockage I've been discussing.
5. You didn't show your data logging consumer and I'm starting to suspect that's where your root problem is. When you receive data there, do you send it to a file writing function that takes a *path* or one that takes a file *refnum*? You should only open the file with a path input one time and all the consumer writes should go to a streaming writer function that takes a refnum.
[Edit: Actually, your attempt to update 8 graphs with a total of 8000 points at 50 Hz might just tax your overall CPU enough to bog down your producer due to CPU starvation. You could explore this by adding a further timing "throttle" to your display consumer. You're already using a Notifier and accepting lossiness, this becomes an easy way to increase the lossiness to troubleshoot the root cause of the slowdown. You could just add something like a 250 msec wait timer into the loop, probably wiring the output to the case structure border to establish dataflow.]
I think it's the plotting that slows down the monitoring.
Thank you for your reply altenbach!
While I was simplifying the VI, I found the problem I think...
It's the "Recording" value property node in the monitoring loop. When I change it to local variable, the RT Monitoring loop seems to be okay and not being affected by the plotting loop now. I wonder why though. It doesn't seem to have anything to do with the waveform graphs. When I was using the value property node, plotting 1 waveform graph is okay. When plotting 2 or more it starts to slow the RT Monitoring loop down and eventually gives me a "not able to keep up with the hardware" error.
Most likely, the property node forces a synchronous thread switch to the UI thread, while updating the graphs can be done asynchronously at a more leisurely pace while all the code needs to do is push the data into the transfer buffers.
Still, you are unreasonably hammering the UI with gigabytes of data per second. Look a the buffer allocations. Not sure how efficient that array splitting and rebuilding is. Why can't you index out the first element, apply the scaling, then replace it back where it was. (Also, that divide and multiply with the same factor seems a bit silly ;))
Thank you very much for your reply! All your comments are very helpful.
1. I did tried 5000 samples per iteration. It improved the situation, but after running for a while it still jam up and gave me the "not able to keep up with hardware" error.
2. I do have a fixed size for the queue. My strategy here is when monitoring I use lossy enqueue, which will drop the elements when there's no space. When recording, I use regular enqueue and hope the logging loop to keep up and so I won't lose any data. It has the potential risk of bogging up and cause error, but if I don't plot the data lol everything ran nicely and the logging was very fast I kept track of the loop time. I like your suggestions and thank you for them. I think I will try the infinite size and flush out method since I don't want to lose data.
4 and 5. Thank you. Although I think I found the problem as the other user altenbach suggested me to upload a simplified version of the VI for everyone to see and debug. As I was simplifying the VI, I removed the "Recording" value property node and the case structure in the monitoring loop. And it worked! I could plot all 8 waveform graphs without slowing down and causing problems with the monitoring loop. When I change the property note to local variable, the monitoring loop seems to be okay too and not being affected by the plotting. I wonder why though. The "Recording" boolean indicator doesn't seem to have anything to do with the waveform graphs. When I was using the value property node, plotting 1 waveform graph is okay. When plotting 2 or more it starts to slow the RT Monitoring loop down and eventually gives me the error.
I used the path to open the files, then write and close them with the refnum. [We also just bought this new computer with relatively better CPU just for the multichannel data acquisition lol Hopefully it holds up!]
I have to admit it is siilly 😅, even though it seems to be working okay at the moment. I was doing some other things in that loop and forgot to change it back to a simpler scheme. Thank you for reminding me that!