Streaming from multiple devices fails if in certain slots

[wired]

Robert Manion loaned me a spare PXI-1042Q chassis so that we could rule out a chassis problem. His chassis produces exactly the same results, so I know it's not a chassis problem (unless it's a chassis design problem). 

[Ed W]

Hi,

Nick and I were able to reproduce this behavior.  We are in talks with R&D about the problem.  I will post back when we have more information.

Ed W.

Applications Engineer

National Instruments

[Jack_A.]

Is there a particular reason that you are using DAQmx events in your LabVIEW streaming program?  At these rates, there's not really a need to use events since there will always be something in the buffer to read out. 

I've played with this some here and I believe the events are what are causing this issue.  Every time an event fires, an interrupt is sent to the controller.  This happens for each device.  When I modified your code to send even more interrupts/events, I noticed that I could get a buffer overflow error all the time.  As I decreased the number of interrupts, things worked better, but it would still sometimes throw the error.  The interrupts also explain the sluggish appearrance of the video since the CPU has to spend time servicing interrupts rather than updating the video.

I have attached some code which streams to disk but does not use interrupts and I didn't see any of these issues.

The one thing I don't have an explanation for is why you were seeing such a big difference in the performance depending on the slot position of the devices.

-Jack

[wired]

Hi Jack,

I am using the events for the following reasons:

1) Reducing the processor load so that I can perform other tasks while waiting for the buffer to fill.

2) To try to 'even out' the amount of data written to the streaming file.  My file protocol requires a means of traversing blocks, and I want to have the block size be fairly constant.  When files get to be 100GB, time spent parsing the file is minimized by having large blocks.

The program was only producing an interrupt each 1/2 second per board - this hardly seems like it should be a problem for the controller.  Also, this method of using events comes from an example provided by NI.  It seems like I should be able to use this capability.

I think the main question to be answered is why things work great (with events) in some slots but not others.  I hope that the final answer from NI is not 'don't use events', which I would find unsatisfactory.  There is a reason why some slots work and some don't, and discovering this reason will be enlightening to us all, I think.

[wired]

Jack,

I looked at your example, and while it is an example of streaming 8 channels to disk, there are a number of differences between our applications.  I want to be able to stream at different rates for each board, therefore I require a separate task for each board.  Also, all of my tasks stream to the same file by passing in a control reference to the log file.

I haven't got time to work with this today, but will be getting back to it on Monday.  I'm hoping we might have more intel on the slot issue soon.

Kevin

[Jack_A.]

Hey Kevin -

I've spent the last two days running lots of tests.  After moving around lots of different slots and by running tests that were ~40 minutes a piece, I didn't found any correlation between test failures and slots dependency.  Out of 100 tests of streaming to disk for 20 seconds, I always got between 8 and 13 failures regardless of slot position of the devices.  The VI that I was using is attached below.  It is very similar to your VI, but I would use a multi device task (put both boards in the task list) so that both boards were running. 

I did make a change recently to the code which was to add the Overwrite DAQmx property which I set to "Do Not Overwrite Unread Samples".  This means that if the CPU does try to read samples soon enough, the board will write the samples to the onboard memory of the device.  Fortunately, these boards have lots of onboard memory.  Out of the 3 different configurations I ran with this property, I never saw a failure.

Have you run your tests without the streaming to disk portion?  I haven't but, my guess is that the VI will run with no errors.  My suspicion is that every now and then, the writing to disks takes longer than normal (maybe the file get fragmented or some other process kicks in) which then causes a buffer overflow error on the DAQ side.

Can you try running the attache dVI and let me know what you find?  Also, you can remove the Overwrite property and test different slot configurations.  If you see the same thing I do, there won't be a link to slot position.

-Jack

[wired]

Jack,

I think you might not be addressing the issue I'm having.  Nick Delbar was supposed to talk to you about this yesterday. Please see the following response I received from Nick a while back via direct email:

******************************************

Hey Kevin,

I just wanted to let you know. I was able to gather all the parts (Sorry, getting the parts together is the hardest part), and I was able to verify that I'm getting the same results as you are. What this shows is it's definitely not than any of your 6120's are broken, or any of the slots are broken. I went ahead, and tried the same slots with some of our M-Series cards as-well, and even though it's not the same symptoms as the S-series cards, I noticed there were some major conflicts depending on the slots of the chassis. I've verified both R&D of the S-Series cards, and R&D on the chassis's, and both teams are looking into this issue, and trying to determine what is causing the issue. I'll keep you updated on what we find out, and I apologize for the inconvenience this is causing.

I just wanted to also mention my test in case you're interested. I just opened up Measurement and Automation Explorer. Created 1 task, and copied all 8 channels into that 1 task. when It was in a good slot. I could sample around 700KS/s just fine. When they were in bad slots. I could only Sample around 75KS/s. The most perculiar part, and I think you mentioned this as-well. When it was in a bad slot, the resources were being used so much, that even the mouse became jumpy, the whole computer seemed to be lagging.

I passed this information on, and also the fact that the M-Series cards were not behaving normally in those bad combinations.

Again, I'll keep you updated, and thanks for working with us to solve this issue.

**********************************************************************

This is the issue for which I'm trying to obtain a solution. I am easily able to stream data to disk at 800KHz using all channels of two separate 6120s when run as two separate tasks (which is a requirement for me), but only when the boards are in certain slots.  Changing ONLY the slot number of one of the devices causes the extremely poor performance.  That is the issue that I hope NI engineers are investigating, because that is the issue I need an explanation for.  My project cannot afford for me to run more tests, unless they are specifically related to this issue, e.g. NI thinks they have the solution and want me to try it out.

Regards,

Kevin

[Jack_A.]

Kevin -

In my testing, I am not seeing any signs of slot dependency on the performance on the acquisition.  In my previous post, I've attached a VI which is very similar to your original one that you posted with an additional property node.  When I run that VI on my system (8186, 2 6120s, 1042), I never had any problems streaming 8 channels from both devices at 800kS/s.  Without the property node set, I would get failures about 8-13% of the time no matter what slots the 6120 were located in.

Attached to this post is another test VI I put together which is also based off of your original one.  This one does use two tasks (one for each 6120) and acquires data with no streaming to disk.  When I ran this VI in multiple slot configurations, I never got an error and was always able to run at 800kS/s on all 8 channels.

I think that some of the original testing that Nick did in MAX isn't really an apples to apples comparison.  I just had him perform his test on my setup and there seems to be enough variability in that test that I'm not convinced that really showed the slot dependent performance you are seeing. 

Moving forward, I'd like to either have you provide me with the exact VI that you are using to reproduce the problem with or have you try some of the VIs that I have posted and see if you still see the same behavior.  What do you think?

-Jack

[Jack_A.]

Oops....forgot the VI.

[wired]

OK Jack,

Thanks for the VI.  I put it on my system and ran it using PXI1Slot5/ai0:3 and PXI1Slot8/ai0:3.  No problem whatsoever - both devices updated their progress in lockstep. This happens consistently.

I changed it to acquire PXI1Slot5/ai0:3 and PXI1Slot6/ai0:3.  When I started it up, the cursor got all jumpy, the Slot5 tasks got jumpy (did not update regularly), and the Slot6 task aborted after a while, getting error -200010. This also happens consistently.

One other interesting observation.  I modified my test application where I'm also doing the streaming to disk to include the property node setting you told me about.  Now, in the 'bad slot' configurations, the following behavior occurs:

If I start a 10-second collection, the cursor gets all jumpy for 10 seconds and the CPU load goes to 100%.  The tasks do not progress at all during this time. After 10 seconds, the Events Received for each task quickly increment from 0 to 20, and each task indicates it is complete.  I went into my file reader, and the data is all there with no gaps!