How to notice callback pileup

[guybrush_threepwood]

Hi, everyone!

I'm not quite sure I know exactly what I'm searching for, but this is my issue:

My device collects data at an external trigger pulse, and plots/handles the incoming data in a callback linked to this trigger pulse (through DAQmxRegisterEveryNSamplesEvent). I'm currently debugging behaviour where the device can't keep up with the amount of incoming data under certain circumstances, and I feel the need to have some visual way of knowing when I get into this situation, like a light switching color or something. I imagine that when I get in this situation then the number of pending calls to my NSamplesEvent callback piles up.

Can I somehow determine the number of pending calls I have, or is there any other method of noticing when my device can't keep up with the amount of data?

[gdargaud]

Off the top of my head, I can't think of a direct method (*), but indirect methods such as CPU use for your program are a rather good indication.Do that by timing the end of your ADC callback and the start of the next. When the two get too close, you have a problem.

There are plenty of ways to mitigate this: bigger buffer, circular buffer instead of just flip/flop, code optimization, adequate multithreading, thread priority, preprocessing (decimation) prior to visualization , etc...

(*) On my custom ADC hardwares the buffers are numbered so as to be sure to know when one is missing.

[RobertoBozzolo]

I can suggest to implement this process in a producer-consumer architecture: a separate thread that handles data acquisition passing back data to the main thread in a Thread Safe Queue (TSQ): data processing should be implemented in the main thread and the amount of data in the queue can be a good indicator of process responsiveness.

This paradigm is demonstrated in a sample program that you can find in <CVISamplesFolder>\samples\utility\Threading\ThreadSafeQueue\Overflow (you can locate it also by searching for Thread in the Example Finder)

[RobertoBozzolo]

Just to elaborate a little on my previous answer, the producer-consumer architecture wants to separate data acquisition activity from processing tasks: the acquisition function (separate thread) should be reduced at the minimum needed to obtain raw data, which are to be passed immediately to the processing area (main thread). This way acquisition thread should be able to sustain data acquisition activities at the pace you need (if it's not, you need to rethink the whole stuff from scratch!). This is a first check to be performed since data acquisition is the core of your process.

Data processing function must be carefully tailored to handle the volume of data supplied by the acquisition thread without loosing any element: most of the suggestions of gdargaud are to be applied in this area: code optimization and preprocessing prior to visualization are the basis to permit developing a stable application.

In this framework, the amount of data left in the queue is not a direct indicator of what you call callback pileup but can be an indirect measure of application performances.