Fourth PCIe-6509 card not detected by MAE

[p_cook]

Summary:

Measurement & Automation Explorer does not detect the fourth installed NI PCIe-6509 card in a HP Z800 Workstation.

Problem Description:

After installing three NI PCIe-6509 cards and booting Windows 7 (64bit), Measurement & Automation Explorer (MAE) shows the three cards.  Microsoft Windows' Device Manager (DM) also shows three "Data Acquisition Devices" labeled "PCIe-6509".  After shutting down the PC, installing a fourth card, and booting, MAE still shows three cards with the same devices names as before, though one of the devices now shows the serial number of the fourth card.  DM now shows four "Data Acquisition Devices" labeled "PCIe-6509".  The three MAE devices report Bus Numbers corresponding to three of the DM "Bus Number" property values.  These bus numbers are 0x3, 0x1C, and 0x60.  The fourth DM has a "Bus Number" property value of "0x28".

To attempt a re-detection of the hardware, I removed all of the cards and removed the Devices in MAE, then reinstalled the cards one-by-one.  I first installed the card which was earlier not reported in MAE.  After installing each of the first three cards, MAE reported each additional device.  Upon installing the fourth card, MAE only reported three devices, with the third device showing the fourth card's serial number.  This is the same behavior as observed initially.

Questions:

Why is Measurement & Automation Explorer not detecting all four cards simultaneously?

Is there a limitation in the driver or MAE which results in a maximum installed card count of three?

Measurement & Automation Explorer System Information

Operating System(OS)                Professional
OS Version                          6.01.7600
OS Info                            
Processor                           Intel(R) Xeon(R) CPU           X5677  @ 3.47GHz / Intel64 Family 6 Model 44 Stepping 2 / GenuineIntel / 3458 MHz
Number of Processors                4
Physical Memory                     3.98 GB of RAM
Drive C:\                           428 GB of 465 GB free
                                   
National Instruments Software:      Version:
                                   
CVI Run-Time                        9.1.0.428
NI-DAQmx Device Driver              9.2.3f1
NI-DAQmx ADE Support                9.2.3
NI-DAQmx MAX Configuration          9.2.3
IVI Compliance Package              4.2
LabVIEW Run-Time 2010               10.0.0
Measurement & Automation Explorer   4.7.4f0
Measurement Studio                  Visual Studio 2005 Support - See individual versions below.
     DotNET                         
          Common                    9.0.20.362
NI PXI Platform Services            2.5.6
NI-PAL Software                     2.6.2
LabVIEW SignalExpress               4.0
NI Spy                              2.7.2.49152
NI System Configuration             1.1.3f0
NI-VISA                             5.0
NI-VISA Runtime                     5.0
LabVIEW Run-Time 8.2.1              8.2.1
LabVIEW Run-Time 8.5.1              8.5.1
LabVIEW Run-Time 8.6.1              8.6.1
LabVIEW Run-Time 2009 SP1           9.0.1

[Frank-L]

Which motherboard do you have, so I can find out if perhaps the problems lies in the motherboard having problems with 4 PCIe devices in at once. Have you ensured that the problem is not the amount of cards in, but either a specific card, or a specific slot that is giving the problem? This can be done by checking each card by itself and then checking each slot by itself to ensure that Measurement and Automation Explorer is not having a problem because of the slot or card, but solely because the PCIe bus is getting overloaded. 

The limit for PCIe devices should not be exceeded by only four devices. The actual limiting factor would be the amount of PCIe slots on your motherboard.

[p_cook]

The HP Z800 Workstation contains the model standard motherboard PCB Rev 1.02.  HP does not list any troubleshooting solutions for this problem.  HP Z800 Workstation - Troubleshoot a problem

Yes, I am sure that the four boards and their four slots are individually functional.  When installing the cards one-by-one, Measurement & Automation Explorer has shown the serial number of the last-installed card, so this diagnostic shows that every slot is working when the cards are installed.

[Frank-L]

The theoretical maximum for PCIe cards in a device is something like 49 PCIe slots on a single computer. We have tested around 17 in our PXIe chassis (which use the PCIe bus) and have been shown to work fine. One thing we can test is to find out which device is communicated to under Dev3 when all four devices are connected. You were saying that Dev3 originally shows the 3rd device, and its serial number, but after placing the fourth PCIe-6509 in the slot you see Dev3 with the 4th devices serial number. It would be good to know which device is actually communicated to in test panels. One suggestion that I heard was that the PCIe bus is not supplying enough power to all of the cards for them to function correctly. The PCIe-6509 has an external power connector that you can plug a 4 pin power supply cable into to ensure that all of the devices are getting power adequately.

Is it a particular slot that is causing the problems? As the specs of the z800 has 6 separate PCIe slots that can be plugged in? Is it a certain card, or always the last card causes problems? 

Lastly, can we try cleaning up Measurement and Automation Explorer in case there was a max database corruption error. What is the Process For Removing MAX Database Corruption?

[p_cook · Accepted solution]

Vary the card insertion order while maintaining the slot filling order (slot 1 -> 3 -> 4 -> 7).  With the initial card insertion order (serial numbers A -> B -> C -> D), MAX reported after each insertion A -> A,B -> A,B,C -> A,B,D.  When repeating the sequence with swapped third and fourth cards (A -> B -> D -> C), MAX reported after each insertion A -> A,B -> A,B,D -> A,B,C.  Thus I conclude the failure is correlated with the slot (insertion order or position) and not correlated to the card.

Connect the disk drive power connector.  Upon connecting power to cards in slots 4 and 7, with cards also in slots 1 and 3, MAX still detects the only cards in slots 1, 3, and 7.

Remove cards.  From a configuration where MAX detects the cards in slots 1, 3 and 7, and does not report the card in slot 4, I removed the card in slot 4.  MAX continued to report the three cards in slots 1, 3 and 7.  Next I restored the four card configuration, and then I removed the cards in slots 4 and 7.  MAX continued to report the three cards, and the self-test of each of the reported cards failed.  This is the first time MAX reported more devices than were currently installed in the computer.

Remove MAX Database.  With cards in slots 1, 3, 4 and 7, I followed the suggested instructions to remove the potentially corrupted MAX database.  For the first boot following the database deletion, MAX recognizes all four cards!  All cards successfully self-test.  Yet upon the next boot, MAX recognizes only the cards in slots 1, 3 and 7, and reports the device for the card in slot 4 as missing/removed (white icon, cannot reset or self-test).  MAX continues to report the card in slot 4 as missing/removed after additional boots/restarts.  This behavior supports the theory that MAX is corrupting its database.  To work around this supposed corruption, the user must delete the database before shutting the computer down and reconfigure the devices upon booting the computer - an undesirable resolution.

Vary slot usage.  I installed the cards in slots 1, 3, 4 and 5 (leaving slot 7 empty instead of slot 5), removed the MAX Database, and booted the computer.  MAX recognized all four cards, as expected from the previous experiment.  Upon restarting, MAX continued to report all four cards as detected.  The table below shows the failing and working configuration, including details of the bus of each slot.  With this working configuration, our software application detects and uses the four cards.

                       MAX 4 Card Recognition
                      ___FAILS___  ___WORKS___
Slot 1: PCIe2  x8(4)   PCIe-6509    PCIe-6509
Slot 2: PCIe2 x16(8)    NVIDIA       NVIDIA
Slot 3: PCIe   x8(4)   PCIe-6509    PCIe-6509
Slot 4: PCIe2 x16(8)   PCIe-6509    PCIe-6509
Slot 5: PCIe2 x16        empty      PCIe-6509
Slot 6: PCI              other        other
Slot 7: PCIe2 x16(8)   PCIe-6509      empty

Thank you, Frank, for your support and expertise.  Together we have found a working setup for our application.