PXIE-5160 fails to initialize properly

[Palickar]

Hello,

I have the following hardware:

PXI-1085 chasis

PXIe-8840 controller

Qty (4) PXIe-5160 500mhz Oscilloscope

So there are 4 5160's in my chassis.  Before I begin taking data, I initialize each of the 4 digitizers and configure them.  Each 5160 has 1 channels, 0 and 1.

Normally, after init and I start taking data, I can see random noise from my UUT, and if my UUT is on, I can see the pulses that the application is designed to measure.  

Randomly, one of the eight total channels will fail to initialize, and it will show a small ~ 4mv DC offset from zero and no signal at all.  Just a perfectly flat line.  The other 7 will be normal.

If I initialize them all again, they may all function normally, or one channel ( no particular one, could be any) will flatline on me.

Every time a new UUT is tested, I init and configure the 5160's, and this bad behavior keeps cropping up.

I have a crude patch in the application to keep me running.  Basically, after init and configure, I look to see if any of the channels are flat DC.  If so, I init and configure again..3 strikes and if I still have a bad channel, I kill the test sequence.

Here is the init and config:

We use LOTS of the older PXI digitizers in our department, and have never seen this before.  We init and configure them in the exact same manner.

Does anyone have any pointers? 

[Palickar]

I have some more info to add.

When we add the NI-scope Reset vi as the second vi after the NI Scope Initialize, the problem gets much worse.  We get a bad channel every time we test our application.

Then we tried less resetting by changing the default RESET=True behavior of the NI SCOPE Initialize VI to false.  Doing this, after system reboot, we always get one channel that is dead.

[Palickar]

So after disabling the reset on the NI Scope Initialize VI, we can run init, and close the handle, then run init again, and use that handle to do what we nned to do , and so far the application works with no dead channels.

This is very strange, as we have many NI racks at out location with all sorts of instruments, and we always just use the default behavior of the instrument initialize vi and get on with our work.

This issue only occurs with the new PXIe chassis and the new 5160 digitizer cards.

[GregoryWF]

Hello!

This is definitely odd behavior and something that we would want to investigate. To start out with, the first thing I would suggest is to create a technical support request at http://www.ni.com/ask so that Support can troubleshoot and escalate this issue as appropriate. Once you've created the technical support request, you can track it through that same link.

There are a couple things we can try so that we have some more information about the problem:

1. If you only used two of the 5160 devices instead of four, do you still see the intermittent 'flatline' issue?
2. If you do not use TClk synchronization, do you still see the intermittent 'flatline' issue?
3. Can you hook up an external oscilloscope to verify the signals coming out of the DUT and see if the 'flatline' behavior exists on the lines themselves?
4. Alternately, can you hook up a signal source of some other kind to all 8 channels and still reproduce the issue?

This will help gather more information to possibly narrow down the root cause of this behavior when troubleshooting with Support. 

[Palickar]

Thanks for the input!  I created a support ticket, here is the link:

https://www.ni.com/my-support/s/case/5003q00001KhOvPAAV/pxie5160-fails-to-initialize-properly

We will get some answers on the posted questions today.

[Palickar]

Here is an update on what I was asked before and from NI Tech Support:

1. The issue only occurs with the new PXIe Chassis and new 5160
1. That is correct.  This is the first test setup that our group built using the new PXIe Chassis and the new 5160 Oscilloscope card.  We are using NI-Scope 18
2. Does the test configuration have the same OS and NI Software?
1. If you are referring to the  other test sets that we built here, this is the first setup with precisely this arrangement of hardware and software.
1. Windows 10
2. PXIe-8840 controller
3. Qty (4) PXIe-5160 oscilloscope.
2. We do have another test setup with Win 7, PXI-1045 Chassis, PXI-8105 Controller, and NI-5152 digitizers.  This setup uses NI-SCOPE 15

1. It is possible to share a reproducible scenario as an example?The scenario as originally stated is very reproducible.  One channel will "flat line" out of the eight once every 5 times the application is run. 

1. If you only used two of the 5160 devices instead of four, do you still see the intermittent 'flatline' issue?
1. I will need some time to do this, as I am competing for time on this hardware with further development efforts.
2. If you do not use TClk synchronization, do you still see the intermittent 'flatline' issue?
1. We must use the TClk sync in our production application.  We are producing a pulse that is on the order of 30 ns wide on a period of about 100 ms.  This pulse is measured by 4 differential devices that are attached to the 8 channels available on the 4 scope cards.  If we do not sync, we cannot see our pulse.
2. That said, I will make a version of our software that does not us the TClk sync, and report the behavior.
3. Can you hook up an external oscilloscope to verify the signals coming out of the DUT and see if the 'flatline' behavior exists on the lines themselves?
1. The flat line behavior was confirmed to be in the test system by an external oscilloscope.  All signals are present on the input to the scope cards during flatline behavior.
4. Alternately, can you hook up a signal source of some other kind to all 8 channels and still reproduce the issue?
1. A signal generator was used to apply a signal to the test set when one of the channels was in the flatline state.  The flat line channel was subjected to a very large pulse input of 8 V that was confirmed by an external oscilloscope and no measurement was indicated on the dead channel in our application.

[Palickar]

This is how we have typically configured and used the digitizers in the past.  We run Dig_ConfigureAndSync.vi.  It calls DIG_Configure.vu 4 times, one for each digitizer in the IVI resources array, and the other snip from DIG_AcquireWfms...vi is how we get the data out.

[GregoryWF]

Thank you for the information! I am going to see if I can reproduce this issue.

[Palickar]

Here's the total contents of the rack:

1 – PXIe-8140

2 – NA

3 – PXI-6528

4- NA

5 – NA

6 – PXIe-4081

7 – PXIe-2527

8 - Pickering 40-153-001

10 – PXIe-6614

11- PXIe-5160

12- PXIe-5160

13- PXIe-5160

14- PXIe-5160

15- NA

16-NA

17-NA

18-PXI-4110

I forgot to add that the Unexpected Behavior Target Labview version is LV 2015, NOT 2015 SP1.

OK, here are the results of the experiment I did last night.

1. Reboot test system
2. In NI MAX, start the task that produces the hardware scope trigger.
3. Run ScopeTest.vi
4. The image below is a screenshot.  There is no signal present on any of the 8 inputs.  Channel X(-) is in the error state.  It will remain in this state until I run ScopeTest again and perform and init and configure again.  This is the scope card in slot 12, channel 1.

SCREENSHOT P1

The next screenshot was produced by running ScopeTest.vi again.  Same situation, no input present on any of the 8 channels, all channels show normal system noise centered on the zero level.

SCREENSHOT P2

Ran this vi 10 more times., all results same as SCREENSHOT P2 above.

Restarted Labview.

Ran ScopeTest.vi

Obtained the following screenshot:

 

Now we see that Z(-) is the  bad channel.  This is the scope card in slot 14, channel 1

This is the behavior that I see in our actual test application, that I cannot post due to corporate restrictions.  In our production application, we init and configure the scopes, and then take reading every trigger, every 100ms.  We confirmed many times that the bad channel does not respond to any signal input and has a small dc offset compared to the rest of the channels.

It is important to note that this problem has been observed dozens of times, and has NEVER presented on the scope channel 0.  In every example so far, all channel failures have been on channel 1.

[GregoryWF]

One question that I'm not sure has been asked yet: After doing the init and configure, what are you using to trigger the measurement to check for the flatline behavior?