DAQmx EveryNSamplesAcqIntoBuffer Event weirdness

[wired]

I wasn't sure where to post this one, because it could be LabVIEW, DAQmx, or PXI, but we'll try it here...

I have a data acquisition application in which I am using four PXI-6120 cards in an 8-slot PXI chassis.  A Star card in Slot 2 is generating the following signals the 6120 board: Sample Rate Clock, Start Trigger.  I have a VI that creates DAQmx tasks in a loop (one iteration for each board), which sets up the acquisition for a given board, e.g. channel list, timing parameters, buffer size, etc.  Each task is committed, and the output of the VI is an array containing each of the Tasks.

I have another VI that, for each task in the list, calls a reentrant data streaming VI which registers the EveryNSamplesAcqIntoBuffer event for the task and then starts the Task.  The VI then enters a loop that contains the event structure set up to handle the event. When the event fires, the data is read from the buffer.

This architecture works well for me up to a point.  I am able to acquire data for all channels of all four boards at pretty high rates (up to 700KHz), but as I continue to increase the sample rate, one of the tasks appears to 'miss' events.  My front panel controls show the number of blocks read, as well as the AvailSampPerChan returned by the DAQmx property node.  Three of the four VIs are reporting the block number in lockstep, i.e. they are all incrementing together, but on of the VIs occasionally pauses for a couple of seconds, then continues.  The AvailSamplesPerChannel that I am reading each time the EveryNSamplesAcqIntoBuffer event is handled does NOT reflect any additional backlog when this occurs!  It's just as if time stopped momentarily for this particular board. I've seen instances where at the end of a five-minute session where I am reading blocks every half-second, three of the VIs will have a block count of 600 and the fourth will be some lower, number, such as 520 or 580.  My input buffer size is only 2 seconds long, so there is no way I should be able to run for five minutes if I truly lost that many blocks. The AvailSampPerChan should steadily increase until I get a buffer overflow error.  Does this make any sense at all?  It Seems like DAQmx is occasionally holding off all activity on the one board, including its sampling of the data!?

[Paul_C.]

Hello Wired,

On first glance, I don't see any quick reason why your missing "blocks" of data on one of the four tasks, but I do have a couple of questions before I speculate on reasons why this is happening.  Are all four of the tasks the same?  Have you been able to isolate the problem to one card?  Did you try the faulty task with one of the other 6120s?  Have you tried doing this without the EveryNSamplesAcqIntoBuffer event and just simply using a DAQmx read to aquire your samples (as shown in our DAQmx examples: Help » Find Examples)?  Also, have you been able to verify what rate over 700KHz this breaks down at? 

Thanks,
Paul C

[wired]

Hi Paul,

I was always having the problem in Slot 3 when the four cards are in slots 3-6. Yesterday I tried placing the 6120s in slots 4-7, and a PXI-4472 in slot 3.  With the 6120s running at 800KHz (maximum rate) and the 4472 running at 1KHz, I was able to acquire data all night with no missing blocks (100000 blocks).  This morning I tried bumping the 6120 sample rate up to 1MHz (using warp mode) and added a 4472 to slot 8, for a total of six cards.  All cards seem to be running just fine in this configuration (no temporary freezups).  The block count is the same for all six cards, as was the case in the overnight run.

I have not tried NOT using the event to trigger the read, because I believe the event-based acquisition should provide better overall throughput.  While these tasks are running, I also need to be able to monitor the waveform of a selected channel in real time and also log the data to a binary file.  I can monitor the data OK (using a queue for each task that I access in my GUI loop), but the logging to file creates a backlog at the higher sample rates.  This is a different issue, though, that I am not concerned with at this point, because I realize what I'm attempting to do is approaching the limits of the LabVIEW's capabilities.  However, the varied performance in the different slots does concern me.  I recently had my PXI controller (PXI-8186) upgraded to the latest revision because of a similar problem (see the following thread)

Streaming from multiple devices fails if in certain slots http://forums.ni.com/ni/board/message?board.id=270&message.id=3718

I thought this had resolved the issue completely, but it seems that it merely improved it.  I have a support call into NI about this, but their request is for me to generate a "simplified version" of the code that exhibits this problem.  It would take me an exceedingly long time to do that, because of everything that this program is doing related to DAQ setup, timing (I'm creating external timing and triggers using a ZTEC card), and other functionality.  I was hoping that someone could describe this anomaly to a developer, to get an idea of what could cause this to occur, e.g. is it a threading issue, a DAQmx issue, a hardware issue, etc.  The fact that the backlog does not show an increase after the pause should definitely be a clue as to where the problem might lie.

[wired]

Paul,

to answer your questions:

Are all four of the tasks the same?  Yes, I am using a reentrant VI (although I have a different VI for each card type), and am calling multiple instances of it simultaneously.  I have six case structures (one for each possible PXI Slot), and each check the card type for a specific slot and calls the appropriate task based on the card type.

Have you been able to isolate the problem to one card?  The problem seems to be slot-related, not card-related. It does not matter which of my cards I put in the slot that has the problem.

Did you try the faulty task with one of the other 6120s?  See the above post.  All cards are capable of running at the maximum rate, depending on the slot.

Have you tried doing this without the EveryNSamplesAcqIntoBuffer event and just simply using a DAQmx read to aquire your samples (as shown in our DAQmx examples: Help » Find Examples)?  See the above post.

Also, have you been able to verify what rate over 700KHz this breaks down at?  There is no specific rate - the problem just gets worse as the rates go up.  I know I have been able to do 500KHZ overnight, but have not performed exhaustive testing on the rates.

[Paul_C.]

Hi Wired,

I have spoken to Brian F. who is dealing with your phone call about this issue.  He is already looking into the situation and he will be contacting you.  He will also post the resolution back to this thread.  

Regards,
Paul C.