Seemingly Random Error with DAQMx, C#, NI-6251 PCIe

[Esteban.R]

Hi JamesDean59

I will like to try to help you and in order to do so I will like to ask you some questions I also wrote a brief explanation of why the answer to that question might help us to find the source of the problem.

Do all the computers that you have with Windows 7 have this problem? I will like to check that all the computer that you have with windows 7 have this problem or only some computer in order to try to find which might be the difference.

Do all the computers have the same NI PCIe-6251 card? I want to know if the problem might be with this specific card or if you have other NI PCIe-6251 cards or other NI cards.

Does the computer with windows 7 is able to run an example code by long periods of time? This will help us determine if the problem is with the card or drivers.

Could run a simplified version of your application to only run the acquisition? This might help us determine if the error in indeed on the section related to the acquisition or if some other step or process of the application might be affecting the acquisition.

Have you check the status of the NI services when you get the problem? Are those still running? When you get that problem are you able to use the card at all or the computer is not able to communicate with the card? Can you use the test panel after getting that error? Could you reset the configuration data of max? This information might help us to determine if the problem is with the drivers or the card.

Last but not the least, could you give me more details about your acquisition task, I will like to get a better idea of what exactly are you doing in your application. (Timing, sampling rate, channels, analog, digital, output, input, etc)

Regards

Esteban R.

[JamesDean59]

Hello Esteban, my replies are below in bold.

Do all the computers that you have with Windows 7 have this problem? I will like to check that all the computer that you have with windows 7 have this problem or only some computer in order to try to find which might be the difference.

Yes--We have two Windows 7 machines and four Windows XP machines. Both Windows 7 machines have experienced this problem. None of the XP unit have.

Do all the computers have the same NI PCIe-6251 card? I want to know if the problem might be with this specific card or if you have other NI PCIe-6251 cards or other NI cards.

Yes--All the machines were identically constructed with the same hardware. Two NI Cards: PCIe-6251 and PCIe-6537. Originally they were all Window XP machines. 

Does the computer with windows 7 is able to run an example code by long periods of time? This will help us determine if the problem is with the card or drivers.

Let me know which of the example code programs you would like me to run and I can see about running it over night.

Could run a simplified version of your application to only run the acquisition? This might help us determine if the error in indeed on the section related to the acquisition or if some other step or process of the application might be affecting the acquisition.

I can look in to running this test. Since I have separated all the acqusition code into a separate space I might be able to do this with relative ease.

Have you check the status of the NI services when you get the problem? Are those still running? 

Yes, the NI services were still running. The computer is not restared during this time nor is the services manager accessed.

When you get that problem are you able to use the card at all or the computer is not able to communicate with the card?

I believe the computer can still communicate with the card. Once the software is restarted all is fine. The computer is not restared during this time nor is the services manager accessed.

Can you use the test panel after getting that error? Could you reset the configuration data of max? This information might help us to determine if the problem is with the drivers or the card.

I can look into this and see if I can still access the test panel after the error occurs. I will also reset the configuration in MAX.

Last but not the least, could you give me more details about your acquisition task, I will like to get a better idea of what exactly are you doing in your application. (Timing, sampling rate, channels, analog, digital, output, input, etc)

I should mention that these tasks were written back in 2008 (long before I was hired) and were functioning without a problem and conitue to work without a problem, with the exception of the periodic communicaitons disconnect when we moved to Windows 7 and newer NI DAQMx drivers.

We have two main acqusition tasks:

1) ANALOG DAQ Card Task:

This tasks collects sample data from one channel at a time. This tasks sampling rate and quantity of samples varies with respect to the speed at which our test is running and a few other variables. Sample rate can vary between 1.25 MHz million to 50Khz depending on our encoder under test. Quantity of samples can range from 1 to 19 million samples again depending on our encoder under test. 

2) DIGITAL DAQ Card Task:

The digital task uses the "OneChannelForAllLines" channel grouping and collects one port's width. The sample rate is set to the devices's maximum of 50 MHz. The quantity of samples we take in vary from 45 million samples (slow speed) and 2 million (high speed) We use this to asses the quality of the output of our devices.

I will work on getting a few of the other questions answered. Hopefully this gives you a good amount of information in the mean time. Thanks Again.

-Kris

[JamesDean59]

---

Clarified a few of the answers, new text is in bold+red.

@JamesDean59 wrote:

Hello Esteban, my replies are below in bold.

Do all the computers that you have with Windows 7 have this problem? I will like to check that all the computer that you have with windows 7 have this problem or only some computer in order to try to find which might be the difference.

Yes--We have two Windows 7 machines and four Windows XP machines. Both Windows 7 machines have experienced this problem. None of the XP unit have.

Do all the computers have the same NI PCIe-6251 card? I want to know if the problem might be with this specific card or if you have other NI PCIe-6251 cards or other NI cards.

Yes--All the machines were identically constructed with the same hardware. Two NI Cards: PCIe-6251 and PCIe-6537. Originally they were all Window XP machines. 

Does the computer with windows 7 is able to run an example code by long periods of time? This will help us determine if the problem is with the card or drivers.

Let me know which of the example code programs you would like me to run and I can see about running it over night.

Could run a simplified version of your application to only run the acquisition? This might help us determine if the error in indeed on the section related to the acquisition or if some other step or process of the application might be affecting the acquisition.

I can look in to running this test. Since I have separated all the acqusition code into a separate space I might be able to do this with relative ease.

Underway. I have created a small program with a thread with a loop that acquires a few channels of analog data, calcualtes the average voltage and display the result to the screen.

Have you check the status of the NI services when you get the problem? Are those still running? 

Yes, the NI services were still running. The computer is not restared during this time nor is the services manager accessed.

Verified. All NI related services were still running when the exception was thrown.

When you get that problem are you able to use the card at all or the computer is not able to communicate with the card?

I believe the computer can still communicate with the card. Once the software is restarted all is fine. The computer is not restared during this time nor is the services manager accessed.

NI MAX can still communicate, run test panels, etc with cards once this exception occurs

Can you use the test panel after getting that error? Could you reset the configuration data of max? This information might help us to determine if the problem is with the drivers or the card.

I can look into this and see if I can still access the test panel after the error occurs. I will also reset the configuration in MAX.

NI MAX can still communicate, run test panels, etc with cards once this exception occurs

Last but not the least, could you give me more details about your acquisition task, I will like to get a better idea of what exactly are you doing in your application. (Timing, sampling rate, channels, analog, digital, output, input, etc)

I should mention that these tasks were written back in 2008 (long before I was hired) and were functioning without a problem and conitue to work without a problem, with the exception of the periodic communicaitons disconnect when we moved to Windows 7 and newer NI DAQMx drivers.

We have two main acqusition tasks:

1) ANALOG DAQ Card Task:

This tasks collects sample data from one channel at a time. This tasks sampling rate and quantity of samples varies with respect to the speed at which our test is running and a few other variables. Sample rate can vary between 1.25 MHz million to 50Khz depending on our encoder under test. Quantity of samples can range from 1 to 19 million samples again depending on our encoder under test. 

2) DIGITAL DAQ Card Task:

The digital task uses the "OneChannelForAllLines" channel grouping and collects one port's width. The sample rate is set to the devices's maximum of 50 MHz. The quantity of samples we take in vary from 45 million samples (slow speed) and 2 million (high speed) We use this to asses the quality of the output of our devices.

I will work on getting a few of the other questions answered. Hopefully this gives you a good amount of information in the mean time. Thanks Again.

-Kris

---

[Esteban.R]

Hi Kris

So if you have run a small program with a thread with a loop that acquires a few channels of analog data, calculates the average voltage and display the result to the screen, I think that the problem is not neither the driver nor hardware. In other words we need to debug the application that was not program by you. As you did not program this code it might be a difficult task and it will be up to you to keep troubleshooting the acquisition task or reprogram the task.

In case that you want to keep troubleshooting this code here are some thoughts:(the imasges are just examples)

1)      Is the same DAQmx driver version that was installed on the computer with windows XP?

2)      Take a look to the following link to chek if you have the .NET support

3)      Check the target framework

4)      Check that the reference match the version of the driver that you have install.

5)      The name of the variables involve on the application

6)      Paths of files, libraries, etc, that might have change from the windows XP version to the Windows 7

Regards

Esteban R.

[JamesDean59]

I was about to do some debugging on my one Windows 7 machine today and this happened:

I tried to open MAX itself and it refused. It generated a log and dump file which I have attached.

I was able to uninstall the entirety of the NI software and perform a reinstall from the 7.0 installer. This remedied the problem. I wonder what, if any, ramifications this has/will have on the disconnect issues I've been having...

[Deagle]

JamesDean59,

Yes, now that you've reinstalled NI software, I'm interested to see if you receive the same error when running a more basic, smaller program. Can you test this and let me know if you still see the 0x80040311 error?

[JamesDean59]

Good morning everyone,

I had the opportunity to do some extended cycle testing on Friday and over the weekend.

Earlier in the week I extraced the NI data acqustion aspects of code from the main program and created a smaller program that simply acquired analog voltage data, computed averages, then collected digital data and computed some items based on that data. I had this collection and analysis routine set inside of a thread and just looped it until an exception occured. I believe I got to about 4000 test iterations before I stopped the test as the machine was otherwise needed. This leads me to believe that the problem is not within the National Instruments DAQMx assemblies. To be 100% sure I should probably let that test run over a weekend but 4,000 iterations is a pretty good indicator.

So this leads me to believe the problem has something to do with Windows 7 and the .NET framework.

On Friday, I had access to two machines. One Windows XP and one Windows 7. I put the same version/build of the test software on both and let them run.

The Windows XP machine did not encouter a single problem. It completed 425 test cycle of data collection and analysis. I couldn't let it run over the weekend as the machine was needed today. However I was able to run the Windows 7 machine over the weekend and it encoutered a number of exceptions as I expected it would. I was trying to get a grasp on frequency of exception thinking that perhaps there was something there.  I've attached my findings. In red is the amount of time that had passed since starting the test run. I dont see any real pattern or predicitabilty to the exception.

On Sunday, I found that the exception had been thrown from the previous round of test. So I restarted the software and began testing again. About an hour later it threw another exception and stopped testing. I didnt check the machine again until this morning. However what I discovered that the machine had gone to sleep. I woke the machine up, didnt see the exception (it was behind the testing window) and I resumed testing. And it worked. Normally if I try to resume testing once the exception occurs...it just throws more exceptions. So SOMETHING restored the broken connection between our software and the NI MAX server. Either the act of the computer waking up from a sleep cycle OR the amount of time that had passed since the exception occured resolved the problem. While this doesnt give me any clear answers, it is certainly interesting. Since resuming testing the machine has so far accumlated an additional 57 cycles and is still going.

[Deagle]

JamesDean,

What version of the .NET Framework do you have installed on each computer? In addition, can you post some of the code from your minimal example, so we can get a better look at the DAQmx calls. There are some issues caused by computers going into Sleep mode, but my thoughts are that it's the length of time that caused the connection to be reestablished. You can troubleshoot this by manually putting the computer into Sleep Mode after an exception.

[k-space]

Did anyone figure out what exactly was causing this? 

I have a system in the field that is acting in a very similar manner.

Jason

k-Space

jwilliams@k-space.com

[JamesDean59]

Hi Jason,

I just did a cycle test in my software last week and the issue has apparently dissapeared. I was able to run out testing routine continuously for over 48 hours (signifnactly longer than we've been able to in the past) with no exceptions, no disconnects, anything.  48 hours of continuous testing repesents nearly 2 months of normal production testing.

I'm not sure if was something I did or something NI did.  I've been making upgrades within out software on and off for quite some time now. I also did a number of driver upgrades too, currently running 9.8.0f0.

Time will tell though. Hopefully it is actually gone for good. Sorry that I could not be of more help.