PXI

cancel
Showing results for 
Search instead for 
Did you mean: 

PXI-8119 controller crashing / halting

This problem has been going on for about 6 months.  The PXI-8119 is running windows 7 in a 3 PXI rack configuration.  There are 5 occurrences of the controller just shutting off in the last six months, and recently, it has happened about once per week.  The automated tester that it is used on runs 24 hours a day, 6 days per week.  Here is what I find when the controller goes down:

  1. VGA monitor is black, no signal.
  2. Chassis power light, the one on the power button is ON and green.
  3. Power OK / Fault light is flashing green.
  4. No other lights on the 8119 are on, like the network activity or HD lights.
  5. PXI-8364 bus extension has a solid orange light.
  6. Front panel reset button has no effect.
  7. I turned off the rack using the rack power button, and turn the rack back on.
  8. The controller boots to the "Windows Error Recovery" screen we all know..cmd prompt, safe mode, bit normally, etc.  I choose "boot normally"
  9. Windows starts, all looks OK.
  10. When I look in the windows system log, there are no error messages from the time when the PC stopped.  Did get the "Unexpected system shutdown" message.
  11. I rebooted the controller and ran the built in diagnostics.  Memory diag and HD diags both completed successfully.

We have a power monitor on the systems, and I am going to look at the log to see if there is a time correlation with any power abnormality.

 

There is a bit more history with this controller, but it might be all noise.  IF anyone has any ideas, I am listening.  If no one has anything to try i will start adding the history.

0 Kudos
Message 1 of 7
(2,596 Views)

Per the user manual for the 8119, a flashing green power/fault light on the controller itself may indicate a power supply issue.  The manual states that the power supply is not within nominal values.  Hope this helps.

Message 2 of 7
(2,568 Views)

That's good, I missed that part in the Front Panel Features section of the manual.  Now is a good time for the history..

 

This 8119 controller that is having the issue was installed to replace an 8119 that failed in service.  The one that failed was totally dead, no lights, response, nothing.  I took the hard drive out of the failed controller and installed it into a new 8119 controller, which is the one that this post is about.  In the Windows System logs, I can see that the controller ( the one that died) was stopping unexpectedly for 6 months, and one of our techs was restarting it without informing Engineering.

So, two different controllers, both stopping unexpectedly.  They have the same hard drive, and are installed in the same chassis.

Note that the green light on the chassis should blink red if the chassis detects a power fault.  We do not see this.

 

In order of decreasing probability, I am thinking the following root causes:

  1. Intermittent Electrical Power problem in the room
  2. One of the cards in the PXI rack is bad ( would be strange, all cards operate and pass self check)
  3. The PXI rack is bad
  4. The new 8119 that I pulled out of the box is bad.

None of these are any fun, considering that this system HAS to run pretty much all the time.

0 Kudos
Message 3 of 7
(2,563 Views)

Palickar,

 

If your company has a Service Contract, I would recommend contacting our Applications Engineering team to troubleshoot this issue.  You can call into our customer support line at 866-275-6964 or create a service request at www.ni.com/support/ with the Service Request Manager

 

Tyler O.
Applications Engineer
National Instruments
0 Kudos
Message 4 of 7
(2,556 Views)

Palickar, 

 

I agree with your 4 potential causes and the fact that they're in decreasing probability. Intermittent electrical issues are an incredibly common problem, but having a UPS for your rack is a common preventative solution. Are you already doing so, and if not, is there a constraint? 

 

 

Message 5 of 7
(2,398 Views)

Thanks Danny,

 

I just checked the syslog of the system, and there were no further error events since my last message.  All 3 NI racks of this system are on a UPS, and I checked the configuration of the UPS as far as monitoring, trigger points, and power quality.  Everything is configured as it should be for protecting this type of equipment.  We were using a nice digital chart recorder to monitor the power on the UPS outlet powering the chassis, and aside from a few little blips of a couple volts over a period of more than a month, I did not see anything.  So far, I have not  captured a processor shutdown with power monitoring to correlate a fault.  Our plant is pretty large, and was built in ~1950, and there is a machine shop in an adjacent building.  As far as I know, there could be some large machine that disrupts or spikes the power on my test system that is not being used right now.  So I am just waiting for the monster to rear its head again.

0 Kudos
Message 6 of 7
(2,390 Views)

Hello again!

I monitored the power on this controller for a month, and we had NO more incidents at all.  I stopped monitoring in November 2018, and all was fine until 2 weeks ago!

 

Same thing, controller stops, and the Power OK / Fault light is flashing green.  Happened twice in the last 2 weeks.  No errors in the sys or app logs.  Just the "Unexpected shutdown at ..." log message on boot up.

 

We are going to replace the UPS on on 7/19/2019 because we can.

Does anyone think we might need to swap out the PXI rack?

0 Kudos
Message 7 of 7
(2,189 Views)