12-12-2023 08:59 AM
Hello,
I am having trouble with my cRIO-9040. The controller will periodically freeze when running a simple application. It is not only the RT application that freezes - the entire linux kernel becomes unresponsive and I am forced to power cycle the device to regain control. Sometimes this happens after only 10 minutes, and sometimes it runs for 1-2 hours, but never more. My program controls a chemical process so I am reading temperature, flow, and pressure data (from a mix of c-series modules and MODBUS (RS485) devices) once a second and displaying the data in an embedded-UI user interface.
While investigating this lock-up I found that I can also reliably cause the crash to happen by simply displaying the index of a while loop to the screen at 100Hz or faster:
This is my first cRIO project with the embedded UI, so it is possible that I am making a silly mistake. I would understand if the vi just failed to work properly, but crashing the whole linux kernel is a bit of a surprise. The actual program (that also crashes linux after an hour or two) is attached. Does anyone have any suggestions?
Many thanks!
Solved! Go to Solution.
12-13-2023 05:39 AM - edited 12-13-2023 05:50 AM
This sounds weird. I have done much more complicated applications using the integrated UI on both 903x and 904x controllers with several dozen top level VIs running potentially in parallel and displaying their front panel, so if your simple loop already crashes after less than an hour, something is really wrong.
Possible actions in about this order:
1) Do you run 2018SP1 with latest fixes? If not, I definitely would recommend installing that, both for the LabVIEW IDE and the Realtime extension. For the FPGA Toolkit there is no SP1 release.
2) Try to completely wipe the cRIO controller and reinstall the default CompactRIO base image from NI Max. Don't customize this for the time being!
3) If this doesn't help do you have access to a newer LabVIEW release such as 2020 or newer? If so try with that.
4) If all of these doesn't help I would strongly suspect a hardware problem that would require a lengthy and rather painful RMA procedure.
That all said, I'm pretty conservative in updating user interfaces at high update rates. The basic principle is, that the human eye can't really see more than 25 images per second, and if you want something to be intelligible to a human it needs to be even slower. So my loops hardly if ever update user interface elements at every loop, even on Windows, but even much more consistently on an embedded target. It simply makes no sense to tax the OS with things that a user can't possibly take in anyhow. It's part of defensive programming I suppose.
12-13-2023 08:49 AM
Status LED's show anything abnormal?
Did this problem just start to show up in the setup?
Abnormally warm ambient temperature area or cRIO running hot?
-AK2DM
01-17-2024 03:09 AM
Rolfk,
Many thanks for replying, and apologies that I did not reply sooner (it seems I did not have my notifications set correctly). I have been working with an NI technician on this issue. Your suggestion to try a different labview version seems to have been the most on-point. I was running labview 2023, using the RT-Linux system image Q1 (I attached my code in LV2018 only to be sure most people could read it). The technician confirms that he sees the same behavior on his controller when running the same labview/RT-linux versions, but does not see this behavior in labview/RT-linux 2021. I have switched to the RT-Linux system image 2023 Q4 and this appears to be better, but I cannot yet say if the problem is completely solved in this version.
I agree with you about the fast updating - my actual application only updates the front-panel elements once a second. The code snippet that I pasted (which updated 100x a second) was merely the most efficient way that I found to provoke this freeze-up on my controller. I made it for debugging purposes. More realistic applications required between 2 hours and 1 day to cause the freeze-up, but this one would freeze my controller in 10 minutes, allowing for faster iteration.
I will post again when I have a more firm resolution to this issue.
Best regards
01-18-2024 02:00 AM
Hello Everyone,
I received confirmation from NI technical support that this behavior is only present in the cRIO 2023 Q1 - Q3 drivers. It has been fixed in the Q4 release, and it is also not present in earlier versions (multiple versions of LV 2021 were tested in detail). Now I can move on to actually deploying my application!
Cheers