DAQmxBase Acquiring at high frequency

[JoeFriedchicken]

---

But as I have just said, every things run well with NIDAQmx on RHEL4/5 ... even thought it's not supported.

Does someone know in which circumstances the kernel stack overflow happens ?

This could help us make more test on the reliability of our application ...

Actually it's quite simple, it can be summarized as an infinite acquiring task.

---

Hi Atos,

Thanks for sharing your history. I would like more clarification on what problems you see when you run DAQmx on RHEL 5. You mentioned a 'kernel stack overflow' but this is a different problem than the one you have with DAQmx Base (RLP Invoke Node <err>There is <b>not enough data</b> in the DMA buffer to fill the VI's read buffer.  Either request less data, or wait until there is additional data in the DMA buffer.)

What happens when you run your task on RHEL 5 with DAQmx 8.0?

[Atos]

Hi Joe,

I will clear the problem to avoid misunderstanding :

+ When I use DAQmxBase on RHEL5, I get the error above (RLP Invoke Node <err>There is <b>not enough data</b> in the DMA buffer to fill the VI's read buffer.  Either request less data, or wait until there is additional data in the DMA buffer.)
+ Using DAQmx on RHEL5, runs well for the moment. But I know that NI do not support DAQmx on RHEL5 because of a possible kernel stack overflow.

Our application is designed to provided signal acquisition in nuclear power plant. It's better for us to use a driver which is compatible with our OS.

As we have seen before, DAQmxBase don't allow us to acquire at hight frequency, so we take the risk to use DAQmx instead.

Here is my question :

---

Does someone know in which circumstances the kernel stack overflow happens ?

This could help us make more test on the reliability of our application ...

---

Thanks again for your help, 

Sylvain 

[JoeFriedchicken]

Hi Atos,

Thanks again for clarifying 🙂

The DAQmx Base error you're observing is an internal bug, and a corrective action request has been created for it (CAR 125793). Basically, at high DMA rates (so for fast analog input or analog output tasks), the driver is unable to correctly count how many samples have been moved off the device. Rather than continuing in uncertainty, it throws this error.

The kernel stack overflow phenomenon comes from how the kernel was compiled. As Shawn points out [1]:

Lastly not only does the Linux kernel move at a rapid pace, but each Linux distribution can modify, and configure the kernel in different ways.  The two distributions you asked about (FC 5 and RHEL WS 4) have a 4k kernel stack.  Most distributions have an 8k kernel stack.  In our internal testing of NI-DAQmx 8.0 on RHEL WS 4 we could cause the kernel stack to overflow.  This leads to a system crash, and is why NI-DAQmx 8.0 does not support RHEL WS 4.  A new version of NI-KAL will not fix this.  As was previously mentioned if you want to use one of these distributions it is recommended that you recompile your kernel with an 8k kernel stack.

So the condition that causes the kernel stack to overflow is an insufficient kernel stack size.

Obviously, you will need to test the stability of the unsupported Red Hat release by mimicking the conditions you expect to encounter. Since there isn't a way to use DAQmx Base right now, your decision appears to be use an older computer so you can use DAQmx 8.0 on a supported Red Hat release, or test DAQmx 8.0 on an unsupported Red Hat release on a newer machine.

Your internal processes and resources will ultimately determine which is best for you, but given that you're trying to monitor a power plant, you hopefully have enough to make that choice.

[1] PCIe-6259 Linux kernel 2.6
http://forums.ni.com/ni/board/message?board.id=250&thread.id=22537

Message Edited by Joe F. on 09-23-2008 09:27 AM

[Daniel Lathrop]

Hi Joe,

Has this error been solved yet?  I am running NI-DAQmx Base on a Mac with a PCIe-6251 card.  I need to acquire data on a single channel at 1.25 MHz.  I get "Error 42 occured at RLP Invoke Node: Possible reason(s): There is not enough data in the DMA buffer to fill the VI's read buffer. Either request less data, or wait until there is additional data in the DMA buffer." whenever I set the acquisition rate to 200 kHz or higher.  If this has not been solved, do you have any ideas on how to get it to work better?

Thanks,

Daniel Lathrop

[JoeFriedchicken]

---

Daniel Lathrop wrote:

Has this error been solved yet?  I am running NI-DAQmx Base on a Mac with a PCIe-6251 card.  I need to acquire data on a single channel at 1.25 MHz.  I get "Error 42 occured at RLP Invoke Node: Possible reason(s): There is not enough data in the DMA buffer to fill the VI's read buffer. Either request less data, or wait until there is additional data in the DMA buffer." whenever I set the acquisition rate to 200 kHz or higher

---

Hi Daniel,

I did determine the source of the problem and even mocked up a possible fix for it. My initial tests allowed me to sample with M Series devices at full rates (eg one channel @ 1.25M, and multi-channel @ 1M aggregate) in both Windows and Linux. I haven't tested performance changes (for better or worse) on Mac, but I suspect that the situation would be the same.

I'll look into it this week and let you know what I learn. Expect an update by Friday afternoon.

[JoeFriedchicken]

---

Joe F. wrote:

I did determine the source of the problem and even mocked up a possible fix for it. My initial tests allowed me to sample with M Series devices at full rates (eg one channel @ 1.25M, and multi-channel @ 1M aggregate) in both Windows and Linux. I haven't tested performance changes (for better or worse) on Mac, but I suspect that the situation would be the same.

I'll look into it this week and let you know what I learn. Expect an update by Friday afternoon.

---

I built the Mac framework and tested AI DMA performance on a PCIe 6259. The results were similar to Linux and Windows: I was able to read one channel at 1.25 MS/sec for 10 minutes. I stopped the task at that time, and no errors were reported. I was also able to read from two channels at 1 MS/sec (aggregate) for the same amount of time.

I believe this matches your application's requirements, and I am interested in confirming it with you. However, my changes haven't been fully tested and they currently aren't on the release roadmap, so I'm reluctant to post this very alpha framework publicly. If you're still interested in trying the updated library, please reply and grant your permission for me to contact you at your email address (as registered with the NI forums). We can discuss things further offline.

If other readers out there (hello to the future 🙂 would like this library for their Mac or Linux application, please reply as well, and I'll consider each opportunity case-by-case.

[Daniel Lathrop]

Joe,

Please contact me at my email address to discuss this further.

Thanks,

Daniel Lathrop

[cericdahl]

Hi Joe,

I am receiving a similar error, and am wondering if it is at base the same problem. Most frequently I see the error:
Error 42 occurred at RLP Invoke Node
Possible reason(s):
The DMA buffer overflowed because data was not read from the buffer as fast as the DMA channel wrote to the buffer.

I am using a PXI-6221 with NI-DAQmx base on a redhat machine (not mine, so I don't have the specs handy). I can produce the error in the example vi:
"Acq&Graph Voltage-Int clk-Dig Ref" (finite AI sample with pretrigger)
under the following circumstance-

Sampling rate > 170 KS/s (on one channel or aggregate over multiple channels)
Delay of > 0.6 seconds between starting the task and receiving the trigger.

The max delay time seems independent of the sampling rate, and I see no dependence on the number of pretrigger or total samples taken. At 160 KS/s I can wait forever (if I set a large timeout) before sending the trigger. At high sampling rates, the error occurs when the trigger arrives.

Right at the threshold rate (~170 KS/s), I see the error immediately, with a different detailed message:
Possible reason(s):
There is not enough data in the DMA buffer to fill the VI's read buffer. Either request less data, or wait until there is additional data in the DMA buffer.

If the trigger is prompt (within half a second), I can run both above and below 170 KS/s, but not AT 170 KS/s! (I don't care about the exact rate, but I would like to acquire at high speed with pretrigger and up to a several hours between triggers.)

I use Labview Version 8.6,
and NI-DAQmx base version 3.x (not sure how to find out the exact version)

thanks-
Carl

Message Edited by cericdahl on 06-30-2009 06:23 PM

[JoeFriedchicken]

Hi Carl,

It does indeed look like you're hitting the same high-throughput bottleneck in the DMA library as Daniel. Would you mind sharing a bit more information?

1. What RedHat release is the system running?
2. What version of DAQmx Base do you have? (You can find this in the readme file [1].)
3. Are there any specific reasons you're using DAQmx Base to program PXI devices rather than the full DAQmx?

[1] DAQmx Base readme
/usr/local/natinst/nidaqmxbase/documentation/README.txt
Or just issue

head -n1 /usr/local/natinst/nidaqmxbase/documentation/README.txt

[cericdahl]

Hi Joe,

1 - /etc/redhat-release says:

Scientific Linux SLF release 5.2 (Lederman)

(my understanding is that SLF is essentially RHEL)

2- NI-DAQmx Base 3.2.0f1

3- We haven't tried the full DAQmx, since (I think) it's not supported for RHEL 5

Thanks!

-Carl