cpu usage causing file write problems

[mgilorma]

I have four pci-6071e boards in my computer and have been working on a data aquistion program that someone else wrote a while back using visual c++ .net.  The problem is that when I am aquiring data at the highest rate allowed by my software program (1M S/s aggregate per card), the resulting data files for each channel are loosing data.  I assume this has something to do with the fact that in when running at the max rate for the system 4 M S/s, the CPU is maxed out.  From what I have read on this forum that is to be expected when running at higher rates and I'm starting to wonder how I can acheive my goal of acurately aquiring 4 M S/s.    In my testing I have noticed that things get even worse from a CPU standpoint when I run fewer channels at a higher rate.  So I wrote a little test program just to aquire data (not to record it), and I found that running one card with 64 channels aquired at 15625 S/s runs just fine from a CPU standpoint, however if I run one card with 10 channels aquired at 100000 S/s, the CPU is maxed out at 100%.  My question is that even though I am aquiring the same amount of samples in both tests (1M S/s)  why is the latter so much more CPU intensive?  Also, is there an alternative approach that will help me a achieve my desired results?? 

any help would be greatly appreciated.

[Jonathan_Brumley]

Are you using DAQmx?

If so, how many samples does your program read at a time before logging to disk?  If you increase the number of samples you read at a time, (and subsequently write to disk), you can reduce the CPU utilization and increase the acquisition rate.  There is some overhead for each Read that you use.

I suspect the reason your multiple channel data rate is higher is because, with more channels, you are reading more data with each read call. 

Hope this help,
Jonathan

[mgilorma]

Yes, I am using DAQmx.   If I am reading your message correctly, your suggestion is to increase the number of samples I read at a time to decrease CPU utilization.   This however is my exact problem, because when I aquire fewer channels at high rates (10 chans at 100000 S/s) I run into my CPU issues.  If I run more channels at a slower rate (64 chans at 15625 S/s) it runs just fine, with almost no CPU usage.    In both tests the card aquires the same number of samples, but in the first example it is sampling much faster which from what I have read might explain the CPU usage problem, because the card is aquiring data so fast that it never has a chance to sleep??  I'm thinking I'm having the same problem that other people had experienced with the 7.3 drivers??  I am using 8.1 drivers.

[Jonathan_Brumley]

In the case where you are reading only a few channels at a high rate, can you increase your read size?  What I mean by Read Size is the value you pass to DAQmx Read.  (Keep in mind when that the value specified to DAQmx Read is the number of samples that you want from each channel).   What I am asking is: if you are reading 100 samples at a time, can you instead pass 1000, 10000, or more samples?  Increasing the read size should help you keep up.

DAQmx is in many cases limited by how many Reads you can do in one second.  If you increase the rate 10x and you DON'T increase the read size, then you will increase the load on DAQmx by 10x. 

For a PCI device like you are using, my experience is that a DAQmx Read on a continuous task will take a minimum of about 50 us (on my 1.6 GHz computer).  On top of that, there is some additional overhead for each sample you add.  However, there is some minimum time necessary to handle the logic for buffering and to snapshot the hardware.  This is why, your system will generally max out if you try to exceed 10000 calls to DAQmx Read per second. 

Hope this helps,

Jonathan

[mgilorma]

The most reads that I ever call are 4 per second, one for each card, no matter what rate I am running at. Each read size is equal to the Sample rate I have each card set to.  I believe the reason that my CPU is maxed out is due to the amout of work that the driver is doing to keep up with the higher rates that I am trying to aquire at.  Running at 4M S/s with the cards only able to hold 512 samples per channel means that the 40 channels evenly spread across 4 cards are transferring data off the cards around 7840 times per second or 196 times per channel.  If I have the same aggregate rate across 256 channels the driver still has to do 7936 transfers per second or 31 transfers per channel.  So it would seem that running more channels at a slower rate would actually preform worse than fewer channels at a higher rate from the driver standpoint.   Or am I looking at this incorrectly?  I have another thread along these same lines that   I have attached a test program that I have been working with that illustrates my problems.

thanks for all the help, it is greatly appreciated

[Jonathan_Brumley]

Hi again,

I reviewed your application here with a .NET expert - we're not sure what exactly is causing the performance problem, specifically we're not sure why acquiring multiple channels would be faster than acquiring a single channel (at the same aggregate rate); however, here are some ideas for speeding up your application:

1.  First of all, you are using SynchronizingObject, and you may not need to.  More documentation is available here:

    ms-help://MS.VSCC.2003/NI.MeasurementStudio/NINETMain/XML/Events_Callbacks_ThreadSafety.html
    Events, Callbacks, and Thread Safety in Measurement Studio .NET Class Libraries

Basically, setting SynchronizingObject causes your callbacks to *all* get serialized on the UI thread, rather than getting called directly by multiple threads.  Using SynchronizingObject allows you to avoid multithreading programming issues, but it obviously also has a performance impact.  Since you've got 4 callbacks running at the same time, each will have to complete with the others (and, of course, all the other normal Windows messages) to actually get processed.  Since it looks like you're using just streaming to disk in the callback and not examining shared state, why not stop using Synchronizing Object for the DAQmx callbacks?

2.  Instead of using callbacks, you could try starting 1 or more threads (4?) and use synchronous reads.  The callback approach itself has some overhead compared with just calling a synchronous read.

3. You're scaling data in the callback, before writing it to disk.  This is typically not recommended for people trying to stream to disk.  You're going reasonably fast so depending on the performance of your disk, trying to stream scaled data may be too much.  A better approach would be to stream data in compressed raw format and scale later.  There's a .NET 'compressed streaming' example in the DAQmx Professional Toolkit that explains how to do this.

[mgilorma]

Thanks again for responding,

I will look into these suggestions to test their viability.  I was wondering however if your .NET expert or anybody else there could provide me with an example program that would be a proof of concept for my setup? 

thanks

[mgilorma]

hi,

Due to some massive oversights on my part, I realized that I'd better output any NIDaq exceptions instead of ignoring them.  What I found is that I am getting the samples no longer available exception when just aquiring 4 cards with 10 channels at 100000 S/s.  I get the same error when recording 4 cards x 64 chans x 15625 S/s.  So I guess the first thing I need to do is figure out how to aquire 4 cards x 10 chans x 100000 S/s?   I should probably worry about aquiring data before I worry about recording it, any ideas??


thanks for all your help

[MickeyD]

Hi,

I have read through your whole thread and it seems like your still having troulbe with getting the PCI 6071E to acquire at the necessary rate.  Since we are using DAQmx there are couple things we should check out and see if the card is working to max capability.  If you go to Measurement and Automation Explorer can we do this acquisition on one channel?  Test this out by going to Devices and under DAQmx devices right click on a 6071E and try to setup the acquisition for a single channel.  You will probably have to increase the samples to read field to get it to work at 100,00 S/s.  Remeber what number you set this to and then set the samples to read value in your code to this value.  The multiple cards should not be a problem if you have a high end CPU, but it will require more CPU time to run the four cards in parallel, so it is not to be complelety disregarded.  Let me know what behavior you see in the test panels and we can resolve this from there.

Have a great day,

Michael D
Applications Engineer
National Instruments

[mgilorma]

Hi, I believe I have gotten things to work.  I found some documentation on the buffer size that gets set in the driver when you are doing a continouous read (like in my app).  So I used the Task->Stream->Buffer->set_InputBufferSize( int ) function to set my buffer size to 1,000,000 for when I am sampling at the highest rate (100,000 S/s with 10 chans), and everything seems to be working much better.  The CPU still is running at 100% but I am able to aquire with no exceptions and also record the data to the disk.  Hopefully I have not overlooked anything and this will be the end of this thread!

Thanks for the help