I'm trying to aquire sensor data using a NI PXIe-8840 RT.
8 modules are connected to the PXI controller: 6 PXI-6238 boards for measuring current and 2 PXI-6230 boards for measuring voltage.
Each board uses up to 8 analog input ports with sensors attached.
One DAQmx task is created for each board with the following configuration:
sample rate = 10kHz
number of samples read per cycle = 5000
Within the task, one voltage/current channel is created for each analog input (i.e. 8 channels/board)
Each task is connected to a callback event:
retVal = DAQmxRegisterEveryNSamplesEvent(
pointer to callback function,
I'm expecting a callback to be triggered every 500ms (based on sample rate and number of samples read) and it works most of the times, but every ~6h or so a random board seems to be skipping/missing an aquisition somehow.
In the callback function I'm first checking which measurement task that triggered the callback, and then reading the aquired data for all ports into a large buffer.
retVal = DAQmxReadAnalogF64(
(8 * 5000), /* 8 ports with 5000 samples each */
When the fault occurs the return code of DAQmxReadAnalogF64 returns -200279 which tells me:
"The application is not able to keep up with the hardware acquisition.
Increasing the buffer size, reading the data more frequently, or specifying a fix
ed number of samples to read instead of reading all available samples might corre
ct the problem."
This seems to be accurate since I've measured the time between the data aquisitions when the fault occurs (more than 1 second).
But I believe that I can't read any faster, since the callback should be triggered when 5000 values are read. Am I correct?
Currently overriding the input buffer size to 15000 samples, but I'm afraid that it will only hide the problem by not throwing an error message when an aquisition is in fact missed.
Any help is very appreciated since it is a very time consuming task to debug when the fault only occurs every 6:th hour or so...
Thank you (and sorry for the long post)!
No great ideas about root cause, so let's focus on a workaround.
Caveat: I program DAQ devices in LabVIEW and don't know details of any text language API for DAQmx. So I can describe the ideas, but can't write you up any code.
1. Oversize your task buffers. When you configured your sample clock & buffer, let's make them hold, say, 5 callbacks worth of samples, i.e. 25000. This alone might be all you need to do.
2. In the callback function right after reading your 5000 samples, query the task for the # of available samples. This should usually be a very small number and you can proceed the normal way. But if you find it to be >5000, this would indicate that a callback opportunity was missed somehow. So read another 5000 samples to catch up.
+1 for @Kevin_Price's suggestion.
In graphical LabVIEW, I believe this callback is the Every N Samples Acquired in Buffer Event. For my use cases I typically make the buffer between 4 - 8 times the sampling rate; that ensures that error does not occur. I get the same error when I make the buffer too small.
To further expand on @Kevin_Price's suggestion, you can make the number of points and buffer size an even multiple of the disk sector size. (Most disk sector sizes are 512 bytes, which I will assume here). So change your callback to 5120 samples and make your buffer 40960 samples. This is the most efficient way to save and log data. DAQmx really hums when operating in this mode. I always use a multiple of the disk sector size when the total number of samples exceeds 8MSa per seconds. Using multiples of the disk sector size has allowed me to record continuously for days at a time 8 channels at 2MSa/s for a cumulative total of 16MSa per second. Not using a multiple of the disk sector size I would eventually get that error, the timing depended on the computer system I was using.
Thank you Kevin and mcduff! Next up I will try out your suggestions and update the results in this thread.
Software was running during the night with input buffer size set to 15000 as mentioned in my initial post.
Got no errors from DAQmx, but I noticed that the underlying problem is still present. Sometimes the callback misses an aquisition which may cause synchronization problems when comparing measured data from different sensors.
Currenty running the system without the callback function and instead polling the DAQmxReadAnalogF64 function from the application, just to see if it is the callback itself that causes the problem.
Just to be clear, missing a callback opportunity does *NOT* mean that you'll lose the chunk of data you should have received during that missing callback time. That chunk of data just gets stuck waiting in the task buffer a little longer than it was supposed to.
The next time you do a read, you'll be pulling out that chunk of slightly-stale data. It isn't *lost*. But when this happens, you'll also be leaving the more recent data stuck in the buffer until the next read. That's why I suggested the query for "# available samples" and the possible followup read to retrieve them.
An oversized buffer lets you recover from an occasional missed read opportunity.
How are you saving the data?
The DAQmx API has a built in logging feature for saving files to TDMS, well written and highly efficient.
If TDMS does not work for your colleagues, I suggest you write a helper program that can convert the TDMS files to their preferred format. This is what I do. The stability and ease of the built in logging feature is too much to pass on.
@Kevin_Price: Yes, I understand that the data itself is not lost, but I appreciate the clarification.
@mcduff: Data is just stored in RAM in circular buffers. Unfortunately, I was not farmiliar with TDMS until now. I googled it and it sure seems like a nice feature. I'm noting it down as a possible future improvement, since I don't have the time to re-write that part of the SW.
As mentioned earlier, I tried to remove the Callback-functionality (not using the DAQmxRegisterEveryNSamplesEvent function at all). Turns out that the software has been running for over 40h now without a single "missed aquisition". I don't really know why it did not work as expected earlier, but removing callback and polling DAQmxReadAnalogF64 manually did the trick for me. Maybe the answer why lies within the answers of @mcduff and @Kevin_Price.
This seems to be the best solution for my application, since it must receive sensor data each ~0.5s.
I appreciate all the help I've got in this thread.