Linux Users

cancel
Showing results for 
Search instead for 
Did you mean: 

Memory leak using NI-DAQmxBase

Hi,

I have a fairly simple voltage sampling program (pure C) that leaks memory.

The problem seems to be in DAQmxBaseStartTask()/DAQmxBaseStopTask(),

as demonstrated by the attached program.

When running this program, `vmstat' shows about 40b/s leaking.  However,

moving the DAQmxBaseStartTask() out of the loop shows about constant

memory usage.  The reason for needing DAQmxBaseStartTask() in a loop is

that I need to measure a single physical AI in both differential and NRSE

mode; I do this by alternating between two tasks.

Am I using the interface incorrectly, or is this a bug in the driver/library?

Thanks.

Download All
0 Kudos
Message 1 of 10
(7,165 Views)

Hi Aaron Carroll,

The documentation for DAQmx Base 3.2 states:

C API:

Using DAQmxBaseCreateTask, a DAQmxBaseCreateXXChannel function, and
DAQmxBaseClearTask in a loop leaks approximately 110 bytes per iteration.
If you pass in a string for the taskName parameter in DAQmxBaseCreateTask,
the memory leak does not occur.

Regards,

Peter K

0 Kudos
Message 2 of 10
(5,153 Views)

Hi Peter,

I'm not sure how that applies here, because I am not using

any of those functions in the loop, and I do pass a string in

the taskName parameter.

0 Kudos
Message 3 of 10
(5,153 Views)

Hi Aaron,

You're right; in my excitement about finding memory leak information in the readme, I hastily pasted that comment .

Do you notice the memory leak if you add in CreateTask/ClearTask?  For example:

// strTaskName is passed into the function

DAQmxBaseCreateTask(strTaskName, &task1);

DAQmxBaseStartTask(task1);

// read data

DAQmxBaseStopTask(task1);

DAQmxBaseClearTask(task1);

Based on what I read from the documentation, that should not leak memory.  If it doesn't, then we can look into why the first case does.

Regards,

Pete

0 Kudos
Message 4 of 10
(5,153 Views)

Hi Peter,

Putting create/clear task in the loop still leaks memory.

It looks like the kernel is leaking the memory, because the

userspace app memory size doesn't increase over time,

and killing it does not reclaim the memory.

Thanks,

   -- Aaron

0 Kudos
Message 5 of 10
(5,153 Views)

Hi Aaron,

I've been poking around this problem a bit over the last two days, and I haven't really reached any kind of solid answer yet.

I agree with you that that `vmstat` reports dwindling 'free' memory as your `leaktest` program runs. However, the program doesn't appear to be eating that memory since the 'VmSize' field in the 'status' file in the process's /proc listing remains constant. In addition, the kernel's memory footprint is stable according to `slabtop`.

I do know that the reason you're not seeing memory being reclaimed after you kill `leaktest` is that you're terminating it with the SIGINT signal, as sent by Ctrl-C, but not catching it so that you could exit the program cleanly. Namely, you're not releasing the resources allocated by DAQmxBaseCreateTask() with DAQmxBaseClearTask().

Joe

Joe Friedchicken
NI Configuration Based Software
Get with your fellow OS users
[ Linux ] [ macOS ]
Principal Software Engineer :: Configuration Based Software
Senior Software Engineer :: Multifunction Instruments Applications Group (until May 2018)
Software Engineer :: Measurements RLP Group (until Mar 2014)
Applications Engineer :: High Speed Product Group (until Sep 2008)
0 Kudos
Message 6 of 10
(5,153 Views)

Hi Aaron,

I've been continuing to look into this, and it does indeed appear that there is a memory leak somewhere in the kernel. While there isn't a workaround at this time, I do have a few suggestions and questions for you.

First, your report.txt.gz tells me you're using a PCI-6229. DAQmx 8.0.1 [1] is the preferred driver for PCI cards on Linux. It has more complete functionality and better performance than DAQmx Base. The only reason to use DAQmx Base on Linux is if you have a USB DAQ device like a USB-6008 or a USB-621x since DAQmx 8.0.1 doesn't support USB devices. Is there a particular reason why you chose to use DAQmx Base with a PCI card?

Second, while I understand your test program was a pathological example, it didn't check the return values of the DAQmxBase...() function calls and thereby ignored any errors reported by the driver. You probably noticed that the average value of your readings was 0V regardless of the voltage on ai0. The reason for this is that the DMA buffer overflowed for every DAQmxBaseRead() call and so the data array was filled with zeros. And the reason that the DMA buffer was overflowing is the task's configuration: the task was set to use on-demand timing and read 20,000 samples for each read call.

The typical use case for on-demand timing is single point reads, while multi-point reads are usually hardware-clocked. Still, DAQmx Base does accommodate multi-point on-demand reads, but only for as many samples that can fit into 8kB (which is the size of the DMA buffer for on-demand ai tasks). For M Series boards, each sample is 16 bits, so the on-demand DMA buffer can hold 4,096 samples. By waiting for 20,000 samples, the DMA buffer never had a chance to be read from before DAQmxBaseRead() went to empty it. There are two ways to increase the size of the DMA buffer and fix this problem: either call DAQmxBaseCfgSampClkTiming() and use a sample clock, or continue using on-demand but increase the DMA buffer size explicitly with a call to DAQmxBaseCfgInputBuffer().

[1] DAQmx 8.0.1 for Linux
http://joule.ni.com/nidu/cds/view/p/id/1194/lang/en

Joe Friedchicken
NI Configuration Based Software
Get with your fellow OS users
[ Linux ] [ macOS ]
Principal Software Engineer :: Configuration Based Software
Senior Software Engineer :: Multifunction Instruments Applications Group (until May 2018)
Software Engineer :: Measurements RLP Group (until Mar 2014)
Applications Engineer :: High Speed Product Group (until Sep 2008)
0 Kudos
Message 7 of 10
(5,153 Views)

Hi Joe,

I did try installing DAQmx, but Base installed and worked without much effort.

Moreoever, nowhere could I find a simple statement of the difference between

the two, so there was no compelling reason to spend time getting DAQmx

working.  Since you've made it pretty clear, I'll give DAQmx another shot when

I have time... thanks

I tried to make the test program as minimal as possible.  My real program

does error checking and does use CfgInputBuffer() to increase the buffer size.

Either way, it should not be possible to leak kernel memory from userland.

I didn't know about CfgSampClkTiming(); I'll look that up, thanks!

Finally, I think I've tracked down the problem.  It looks to be a bug in Linux's

"slub" allocator.  Specifically, allocating large chunks via kmalloc() bypasses

the normal allocation algorithm and instead just returns pages... the kfree()

path for this looks to be broken.

I verified this with a simple kernel module that has a kmalloc, sleep, kfree

loop, so I don't think NI-KAL is at fault.  The bug looks to be fixed in v2.6.29,

however NI-KALv1.10 doesn't build against that kernel so I can't check that

this specific problem is solved.

One workaround is to switch the default allocator to "slab", which works fine

for me on 2.6.28 stable.  Poking around a little in NI-KAL, it looks like a simple

solution would be to avoid kmalloc/kfree for large allocations (perhaps by using

alloc/free_pages() directly) for kernels configured with slub.

Cheers,

       -- Aaron

0 Kudos
Message 8 of 10
(5,154 Views)

Hi Aaron,

In my experience with installing NI drivers on different Linux distributions, the most difficult step is installing NI-KAL. Once that's in, the rest of the pieces go in very nicely. So, if you have a system that accepts NI-KAL, then you should be able to install DAQmx 8.0.1 quite easily once KAL has been installed.

I'm still looking into the kernel memory leak. Thanks for your updates. I've been testing with RHEL 5.3 which has 2.6.18, well before SLAB was replaced with SLUB (which looks like it was first introduced in 2.6.22). I'm noticing kernel memory usage increasing while I run a program based off of your leaktest, so perhaps the problem is still elsewhere. However, with help from Shawn, I've been able to verify that kernel clients of our module aren't leaking memory, so there is a stronger case for pointing at the kernel interface and the way KAL requests memory. At any rate, I have more things that I want to try before I will feel confident in sharing any conclusions.

Joe

Joe Friedchicken
NI Configuration Based Software
Get with your fellow OS users
[ Linux ] [ macOS ]
Principal Software Engineer :: Configuration Based Software
Senior Software Engineer :: Multifunction Instruments Applications Group (until May 2018)
Software Engineer :: Measurements RLP Group (until Mar 2014)
Applications Engineer :: High Speed Product Group (until Sep 2008)
0 Kudos
Message 9 of 10
(5,154 Views)

After a few more tests, I'm fairly certain that behavior I've seen on my system is not a memory leak.

The most helpful test was  a control test over the weekend, where I left the machine idle after rebooting. When I got in this morning, I noticed that the kernel had used the same amount of memory as when I ran my leak test. The daily cron jobs and other system processes contributed to an increased kernel footprint (mainly in ext3_inode and dentry caches) rather than the DAQmx Base leak test.

Since my kernel doesn't have the SLUB allocation algorithm, it won't have the leak that you're experiencing. I'm thankful to have the feedback, though, as it's something we can address as we continue to update NI-KAL.

Joe
Joe Friedchicken
NI Configuration Based Software
Get with your fellow OS users
[ Linux ] [ macOS ]
Principal Software Engineer :: Configuration Based Software
Senior Software Engineer :: Multifunction Instruments Applications Group (until May 2018)
Software Engineer :: Measurements RLP Group (until Mar 2014)
Applications Engineer :: High Speed Product Group (until Sep 2008)
0 Kudos
Message 10 of 10
(5,154 Views)