Multifunction DAQ

cancel
Showing results for 
Search instead for 
Did you mean: 

NI cDAQ-9189 stops responding to Labview code at random intervals

Hello all,

 

I am currently running into a problem where the digital outputs of my cDAQ-9189 stop changing state. 

 

Purpose of system: A test fixture designed to run pumps for a reliability test. This also controls several heaters, level sensors, and auxilary pumps to control the flow/temperature of water and cleaning solution being used to test the pump's long term reliability. The general state diagram is attached. 

 

System overview: 

cDAQ-9189 with 2x 9214 Tc modules (Only one used for this project), NI 9205 AI module, NI 9203 AI module, and NI 9375 DI/O module used as the output.

LabVIEW version 17.0 (64-bit) with 72640K memory allocated. 

Code: See Attached

Hardware: All outputs to the test fixture are controlled through 12V relays with their coils controlled by the NI 9375 output card. The system is a mix of 24V and 240V power output through those relays to control 12 UUT pumps, two different heaters, and auxiliary pumps to move fluid between three tanks. The inputs to the system are level sensors and K-type TCs. 

 

Problem: 

Once in a while the output from the NI 9375 will stop changing state. This seems to happen when the software changes from one state to another, with the software continuing on to the next state while the hardware never stop executing the previous state. I.E: the hardware continues trying to drain an already empty tank after the software has commanded the tank be re-filled with the next fluid. The software is not frozen, and can continue to be interacted with but the hardware does not respond in any way. No error code occurs. The problem occurs at seemingly random intervals. Sometimes it runs for 2 days without issue, other times only a few hours. 

Note: Resetting the cDAQ-9189 from NI MAX resolves the freeze, but requires the software to be restarted and we would like this test fixture to run continuously without direct monitoring. 

 

I don't think this is a software issue as the code continues to function after the I/O freeze. No LabVIEW error code is recieved. I am currently monitoring the software's memory usage to determine if there is a problem with a memory leak or similar issue. So far no problems. 

 

Any advice on resolving the issue or further troubleshooting steps to take?

 

Thanks,

Dylan

 

Dylan Howard
Lab Technologist, Alto-Shaam Inc.
Download All
0 Kudos
Message 1 of 16
(3,781 Views)

Hello Dylan,

 

Does this "freeze" happen with a VI that just updates your digital outputs? Perhaps running a digital output example for a while and checking if it also happens there can give some clue as if the problem relies on the code you have developed, or in the driver/hardware itself.

 

I see that the cDAQ you are using uses Ethernet connection. Do you use an Ethernet hub/router/DHCP server, or a link local connection to your PC? Does the network have a lot of traffic?

 

If you probe the input of the Digital 1D Bool Output Write VI do you see the changes there? Meaning, is the code actually sending the changes, but the hardware is not showing them, or the code itself stops updating the values that is sending to the hardware?

 

Let us know your findings. Warm regards,

Message 2 of 16
(3,751 Views)

Oscar, Thanks for the reply. See my responses below: 

 

Does this "freeze" happen with a VI that just updates your digital outputs? Perhaps running a digital output example for a while and checking if it also happens there can give some clue as if the problem relies on the code you have developed, or in the driver/hardware itself.

 

The program reads analog inputs, performs logic bases on that information, sets digital outputs, and writes to a TDMS file. The code has worked for several days at a time in the past without issue. The crash is a more recent development, leading me to suspect some sort of hardware issue with either my PC or the DAQ. My code is attached to the original post if you would like to see it. 

 

I see that the cDAQ you are using uses Ethernet connection. Do you use an Ethernet hub/router/DHCP server, or a link local connection to your PC? Does the network have a lot of traffic? 

 

The DAQ is currently directly wired to a single PC controlling the test. There is no hub/router/etc between them, and there is no other traffic on that network. 

 

If you probe the input of the Digital 1D Bool Output Write VI do you see the changes there? Meaning, is the code actually sending the changes, but the hardware is not showing them, or the code itself stops updating the values that is sending to the hardware? 

 

The code is in fact sending the changes, and the outputs are not changing. The code is continuing to run, but the hardware has stopped responding. 

 

Additional note: I think this may have been a problem with my PC. I shut off all power to the system and let it rest for a good long time, then restarted the PC, DAQ, and LabVIEW. After that it has been running for a full day without the freeze occuring. I will continue monitoring the process and get back to you if it occurs again. 

 

Thank again,

 

- Dylan

Dylan Howard
Lab Technologist, Alto-Shaam Inc.
0 Kudos
Message 3 of 16
(3,744 Views)

Thanks for the answers, Dylan. Keep us in the loop with your findings.

 

Other options that come to my mind that would be worth checking are:

  • Monitor the amount of samples in the output buffer
  • Check for NI DAQmx updates
  • Force reinstall the software

 

All the best,

0 Kudos
Message 4 of 16
(3,740 Views)

I was able to get the software to run for several days without incident, but it crashed again over the weekend. The failure still seems to be intermittent. I'm going to swap to a different cDAQ-9189 after we re-set the test fixture and see if the problem persists. 

Dylan Howard
Lab Technologist, Alto-Shaam Inc.
0 Kudos
Message 5 of 16
(3,684 Views)

Hi Dylanh,

 

I took a look at your code. Right now everything depends on the last VI not having an error to function properly. Have you looked to see if you have an error when the communication fails? If so what is the error?

 

I would recommend pulling your code a part a little bit. I would make a loop that does nothing but the digital loop. One that does the state machine and one the does the TDMs data recording. I would have you look at using a real state machine system to control your system. You could send updates to the different loops that would cause the digital system to update. This would allow you to recover each loop if there is an error when it is processing. This way you never have to stop the program it would just recover on its own. It will take a little bit of reading to accomplish this but there are examples of the state machine in the examples that ship with LabVIEW. There is also a state machine template that you could easily drop into a vi and start using immediately.

 

The flow of your current program will be problematic because you do not look for errors and recover. If you see an error with the DAQ system you could reset and have it start over and pickup where it left off. It would not be to difficult to accomplish this.

 

Tim
GHSP
0 Kudos
Message 6 of 16
(3,669 Views)

Hi Tim! Thanks for taking the time to reply. 

 

I suppose I could have used a state machine structure instead of a loop with a case structure sub-vi, but the code does not seem to be the issue as far as I can tell.

 

No error code is ever received, even after the cDAQ has frozen and I hit the stop button causing the error handler to report any errors that occurred. Also. the error wire is contained within my "Global Inputs" cluster typedef, and is visible on the main VI at all times. If there was an error during operation it would at the very least appear there and when the VI was stopped it would pop up a dialog box. The error handling did function properly while I was troubleshooting the code as I wrote it. Honestly, I wish I were getting an error code as it would point me in a direction for a solution. Instead, the cDAQ simply stops responding to the VI. The digital outputs no longer change state, and the test fixture continues to do whatever it was last doing until I force the cDAQ to reset with NI MAX. 

 

This error can happen after an hour, or after several days of correct operation. There isn't any pattern to it. That what is making me think the problem is something to do with hardware communication between the PC and the cDAQ. 

Dylan Howard
Lab Technologist, Alto-Shaam Inc.
0 Kudos
Message 7 of 16
(3,663 Views)

Looking at your code you only update the display after the heat and fill logic vi. If the code get stuck in one of the other vi's you would never know there was an error. I know it seems like simple code to do this type of work but I think you would reap the benefits. If you broke the digital into it's own loop you could recover from errors if there where any. If the coms quit working you should be able to detect that. 

 

I would have:

an analog loop that recovers on error

A digital loop that recovers on error

A TDMS loop that recovers on error

A state machine loop that tells everything else what to do.

 

I would use an action engine to store all of the information for each loop and access it when I need to get that information. You should be able to detect where the error is occurring and recover if possible.

 

I would also add code to the system that automatically reserves the cDAQ chassis on code restart. I have had this issues myself. I wrote the code to make it recover and reserve the chassis and I never had that problem again.

 

 

Tim
GHSP
0 Kudos
Message 8 of 16
(3,659 Views)

 

Tim, thanks for your insight. I agree that my code could be better optimized to handle errors, but I positive that none are being triggered. Please see the following: 

 

Looking at your code you only update the display after the heat and fill logic vi. If the code get stuck in one of the other vi's you would never know there was an error.

 

- You are correct. I update the display every 50 mS, once per iteration of the loop. I know the code is not stuck in another VI because I can continue to interact with it, changing to manual control, hitting pause, etc, and see the state change in my display. However, despite the software sending a command, the DAQ never actually responds. In essence, the VI sends a command, and then waits infinitely for the DAQ to complete the commanded task because it never actually happens. While the VI waits for a response, I can continue to interact with it.

 

I.E: if I press pause, the display will reflect the state as "pause" and the "pause" Boolean will light up. The output indicator for the stack light will blink, but DAQ never changes the physical light from solid green to blinking yellow. I know this code normally functions properly as I can execute it any number of times before the DAQ locks up. 

Capture.JPG

 

I can force other errors to be displayed, for instance by canceling the window to create the TDMS address. This automatically prompts a dialog box for the error and ends the program. No such error is detected or reported when the DAQ freezes. The code continues to run normally.

 

I would also add code to the system that automatically reserves the cDAQ chassis on code restart. I have had this issues myself. I wrote the code to make it recover and reserve the chassis and I never had that problem again.

 

- This is basically what I am thinking needs to be done. I would like to find a way to detect that the cDAQ is no longer communicating properly and send it a reset command if that occurs. However, I do not know how to implement this. Is there a good example of how that can be accomplished? What can I monitor that will reliably tell me communication status, and how can I programatically send a reset command to the cDAQ? 

Dylan Howard
Lab Technologist, Alto-Shaam Inc.
0 Kudos
Message 9 of 16
(3,655 Views)