DAQmx read measuring slowly

[grabillkid]

I'm still fairly new to LabView, only been using it about 5 months, so bear with me.  

I'm trying to figure out why the read function in DAQmx is running slower than I am expecting.  I am using a PCIe-6363 card connected to two BNC-2110 connector blocks.  I am running the attached code (Position AOM control_prod_cons) and the producer while loop is taking about 14 ms to execute.  Here's why that's an issue:

This subVI is part of a larger structure that we use in our research lab to run our home-built nanofabrication system.  We are using a computer-controlled stage (move sage task), reading sensor monitors from the stage (sensor monitor task) and controlling a shutter that controls laser power getting to the sample (subVI with lightning bolt).  So four analog outputs and three analog inputs.  The stage is being commanded to move in 5 nanometer steps at a speed of 50 microns per second, giving an analog output rate of 10,000 samples per second per channel (3 of them).  The shutter has a much lower output rate, at most 5 samples per second.  Our goal is to sync the stage movement with the shutter, so we need to read the exact position of the stage using analog inputs, one for each axis, x, y and z, polling as often as possible, though in reality it doesn't need to be more than one read every ms.  

Position AOM control_prod_cons takes in two points in space which we would like to move between and two points where the shutter needs to open/close.  The first frame in the flat sequence does some vector math to get the line which we would like to write going out from the origin to the octant where x, y and z would be positive by adding or subtracting, then doing reflections about axes, if necessary.  The ending x,y and z values (which are now all positive) are summed and passed to later in the sequence.  The sensor monitor coordinates are then read in (producer loop) and transformed in the same way the target coordinates were transformed (consumer loop), then compared to the starting (0) and ending (x+y+z) values and the shutter adjusted accordingly. (For troubleshooting purposes, 'off' is 2.5 and on is read from the text file we input to draw the structure).

All of the tests I've done so far tell me that the producer loop is the rate limiting step.  For a line that is 30 microns long, with the stage moving as I described above, I am getting 45 or 46 reads, which works out to one read (across the three channels) once every ~14 ms, or only 72 samples per second per channel, which is *very* slow for the card, and not acceptable for our application.  Am I missing something?  Why is the producer loop taking 14 ms per iteration?  I tried reducing the write sample rate to 1000 samples per second per channel, but that did not change the read rate, so I'm not sure I can pin this on the card.

I've run the DAQmx read function as an NChannel, 1sample read in a while loop as I am doing in the more complicated VI, reading three analog inputs in a very simple VI that only reads the sensor monitors, and nothing else, and the read rate was 2750 samples per channel (read 5000 samples across three channels in 1800 ms), so I know the card can read that fast.  

I have also attached the higher lying VI that sets up the physical channels, for added information.  I have also attached a VI where the producer/consumer framework is not used.  This while loop also took~14 ms to operate.

Sorry this was so long-winded, just trying to make what I am trying to do clear.  If you have any other questions, please just ask.  Thanks in advance for your help.

[Kevin_Price]

There are multiple candidates for the speed limit.  You've explored the DAQmx Read NChan 1Samp already and found it to be only a minor contributor, fairly negligible for the time being.  So we'll drop that one from the list.

The next likely candidate would be the missing AOM_Voltage_Control.vi.  The way you've structured your benchmark timing in the attached code, its speed is every bit as important as your DAQmx Read speed.  Whether you do both in the same loop or separate them as producer/consumer, you're still waiting for both functions to execute the same # of times.   (Actually, you have additional pre- and post-loop calls to AOM_Voltage_Control.vi).

I think the conclusions you've drawn are too specific for the flawed benchmarking method you've used.  The code where you tried to separate the DAQmx Read from the processing using separate producer / consumer loops *still* has a strong timing dependency between them.  Can you find it?  

   It's the queue that links them.  You've given it a fixed size of 10 elements.  Let's suppose your producer code is fast enough to run at 2750 loops per second (like you reported).  If the consumer is slower, the queue will quickly get backed up to its max size of 10.  Thereafter, the producer loop will get stuck at the Enqueue function until the consumer loop iterates again and makes room.  Both loops will then keep proceeding at the pace of the (slower) consumer.

Try letting the queue be unlimited in size, and temporarily disable or comment out all the code inside AOM_Voltage_Control.vi.  See if you don't get timing that's closer to the 2750 loops per second you got in the stripped down test you did.

It's probably possible to speed it up further if you need to, but let's start by getting clarity on the actual DAQmx Read speed.

-Kevin P

[grabillkid]

Hi Kevin -

I forgot to include the AOM_voltage_control, so it is attached here.  It's pretty light.  I tested it this morning, and if put just that VI inside a while loop and ran the while loop for 50 iterations, it took 40 ms.  So that's less than a millisecond per call.

I also tried letting the queue be of unlimited size, and still only got about 45 or 46 iterations.

I wrote some additional code to try and isolate the problem, which is attached.  This is just the write/read DAQ with the producer/consumer loop.  I ran just this code and am getting 45 iterations out of the producer loop, which is exactly the same as I was getting in the more complicated code.

Thanks again for your help with this!

[grabillkid]

Let me also add that I tried reducing the number of write/read channelsfrom three to one on each channel, and that had no effect on the number of loops.

[Kevin_Price · Accepted solution]

Well I'll be darned.  The "is AO task done?" query seems to be the culprit.  I have no recollection of ever being aware that it was such an expensive query.  I don't know if I've tried doing such a query in a tight loop before, but I'm kinda surprised at the fact that I'm surprised.  I don't know if anything's changed, but in case it's a  new-ish side effect, I'm using the new DAQmx 16.0

An alternate approach could be to use a DAQmx Write property node to query Total # Samples Generated.  That seems quite a bit faster.

-Kevin P

[grabillkid]

I wouldn't have thought that query would be expensive either, which is why I didn't try to troubleshoot that.  I'm using DAQmx 12, so it's likely the same thing.  Just so I know, how'd you figure that out?  Just trying to build my "how to figure things out in LabView" repertoire.

I think I have another way of quitting the while loops, so I'll try later today. I'll let you know if I get more reads after that, and if it works, I'll post that code.

[Kevin_Price]

RE: building the "how to figure things out in LabVIEW" repertoire

1. Troubleshooting code is the software person's version of detective work.  Apply your standard cliches like "divide and conquer" and "eliminate suspects."  You basically work on narrowing your focus by reducing the problem to the smallest, simplest form that still exhibits the troubling behavior.  Most of the thought process isn't specific to LabVIEW, though there are times that LabVIEW experience and knowledge helps discriminate between the more and the less likely suspects.

   That said, here's the route I took:

2. First I rearranged the timing measurements to do a better benchmark.  I don't want to measure the time it takes to configure & clear tasks pre- and post-loop.  I just want to measure the loop.  So I set up the time measurement to start after the tasks were started but before the loop(s) and then to end after the loop(s) but before clearing the tasks.

   Here I confirmed your essential finding -- 10's of millisec per loop iteration.

3. Although you had reported fast loop times when everything but the DAQmx Read was stripped out, I decided to confirm it.  I picked this function first for two reasons.  

    First, it's the most crucial thing your loop does.  Any real solution will *have* to include the Read function, so let's check it out on its own.  

    Second, my LabVIEW knowledge suggested that a call to DAQmx Read would have some extra "auto start" overhead if the task weren't started explicitly before the loop.  Even though by all appearances it *should* have been in an explicitly-started state before the loop, I wanted to make sure.  It's a known cause for a similar symptom, so let's make sure we aren't accidentally in that mode.

   Sure enough, it was plenty fast for me too, in the realm of 10's of microsec per iteration.  I then experimented by *not* calling DAQmx Start before the loop, making the task go through its auto-start overhead on every iteration.  It was much slower, but still only about 1 msec per loop.

4. The only other possible suspects are the Enqueue function and the Task Done query.  Here, lots of real-world LabVIEW experience feeding queues at high rates made me *NOT* suspect the Enqueue function.  So the next thing I tried was to add the Task Done query to the DAQmx Read from #3.  And bingo, there it was.

In this particular case of troubleshooting, my many years of LabVIEW and DAQ background didn't really save me a lot of time identifying the culprit.  I checked out 2 of the 3 suspects, exactly what you'd expect if investigating at random, with no prior knowledge.  It's more typical for prior knowledge to be a tangible help, but it's also far from guaranteed.

-Kevin P

[grabillkid]

I had been troubleshooting by breaking down into smaller problems, so it's good to know I was on the correct path!

The timing in the VI's I posted was just one of the many iterations I used with the tick function.  I had also checked exactly what you had in point 3., to see if calling DAQmx was taking a long time.  I incorrectly assumed the property node would be fast, so i didn't even check it.  I guess in the future, I'll check *every* possible function.

[JÞB]

Nice work Kevin

Now, lets get rid of that p-node and drop in a "DAQmx Wait Until Done.vi" with timeout set to 0 instead.  The internal CLFN of that vi might just be a whole lot faster.  Read the help! you will need to find a way to ignore the expected errors thrown when the task isn't done and exit on a no-error condition when the task is complete.

Spoiler

Optomizing that DAQmx Read (NChan 1Samp 1D).vi  Replace the Start Task.vi with a Control Task.vi wired to a "Commit" constant to minimize DAQmx state transitions.  Prior to the "Start Task" the Task is in a "verified" state and needs tor transition all the way back to that state when the single sample is read and the next loop iteration has to reserve, commit and start the task all over again. See Here for lengthy discussion on state transitions.  The Start task may actually commit and then run the Task with the first read merely pulling the already-been-buffered first sample and ending back in commit-  the help isn't super clear on that detail but, IMHO the control task explicit to commit is cleaner and clearer about where the state transitions happen.

[Kevin_Price]

Couldn't resist trying it out.  Turns out "DAQmx Wait Until Done.vi" with a 0 timeout seems to run with the exact same slowness as querying the Task Done? property node.  I kept getting only 60-some loops/sec.

With only the NChan1Samp Read, task in software-timed on-demand mode, I would get roughly 175000 loops/sec.  That was surprising (and impressive!).

I tried getting clever with a hw-timed task, setting Read properties "RelativeTo" = "Most Recent Sample", and "Offset" = -1.  The thought was to let the hw be doing conversions in the background, then querying the task only to retrieve data that's already waiting.  I figured this would be the fastest of all possible options.  Nope --  pretty fast, with 155000 loops/sec, but not the fastest.

BTW, the task was committed and started before starting the looping & time measurement.

-Kevin P