Crio 9039 RT Crashing Periodically problem

[Wang_Liming]

dear all

We have RT Crashing Periodically problem, we fail to solve the problem, try to find help from you.

LV version 2014 sp1 with some 2015 software package

run RT.rexe in Crio9039

#OSName: Linux
#OSVers: 3.14.40-rt37-3.0.0f1
#OSBuild: 200232
#AppName: lvrt
#Version: 14.0.1

During running the RT program in Crio9039 RT Crashing Periodically. The RT will restart automatically.

path: \var\volatile\log\auth.log file suggest the RT reboot crash and reboot cause of session closed for user lvuser

2018-02-10T21:35:01.000+08:00 NI-cRIO-9039-01B76F67 crond[23159]: pam_unix(crond:session): session opened for user root by (uid=0)
2018-02-10T21:35:01.000+08:00 NI-cRIO-9039-01B76F67 CROND[23159]: pam_unix(crond:session): session closed for user root
2018-02-10T21:38:09.000+08:00 NI-cRIO-9039-01B76F67 su[1984]: pam_unix(su:session): session closed for user lvuser
2018-02-10T21:38:09.000+08:00 NI-cRIO-9039-01B76F67 su[23238]: Successful su for lvuser by admin
2018-02-10T21:38:09.000+08:00 NI-cRIO-9039-01B76F67 su[23238]: + ??? admin:lvuser
2018-02-10T21:38:09.000+08:00 NI-cRIO-9039-01B76F67 su[23238]: pam_unix(su:session): session opened for user lvuser by (uid=0)
2018-02-10T21:40:01.000+08:00 NI-cRIO-9039-01B76F67 crond[23531]: pam_unix(crond:session): session opened for user root by (uid=0)
2018-02-10T21:40:01.000+08:00 NI-cRIO-9039-01B76F67 CROND[23531]: pam_unix(crond:session): session closed for user root

after reboot /var/local/natinst/log/lvrt_14.0.1_lvuser_cur.txt show some strange waring

InitExecSystem() call to GetCurrProcessNumProcessors() reports: 1 processors
InitExecSystem() call to GetNumProcessors()            reports: 4 processors
InitExecSystem()                                      will use: 1 processors

<DEBUG_OUTPUT>
02/10/18 下午 09时38分15.066秒
DWarnInternal 0x00000000: CPU information collection failed! Assuming minimal functionality.
/builds/penguin/labview/components/CPUInfo/trunk/14.0/source/lib/CPUInfo.cpp(748) : DWarnInternal 0x00000000: CPU information collection failed! Assuming minimal functionality.
$Id: //labview/components/CPUInfo/trunk/14.0/source/lib/CPUInfo.cpp#2 $

</DEBUG_OUTPUT>
*** Dumping Bread Crumb Stack ***
#** Loading: "/home/lvuser/natinst/bin/startup.rtexe/new_Crio9039 RT 4ch Program.vi"
*** End Dump ***

<DEBUG_OUTPUT>
02/10/18 下午 09时38分15.066秒
DWarnInternal 0x00000000: CPU information collection failed! Assuming minimal functionality.
/builds/penguin/labview/components/CPUInfo/trunk/14.0/source/lib/CPUInfo.cpp(748) : DWarnInternal 0x00000000: CPU information collection failed! Assuming minimal functionality.
$Id: //labview/components/CPUInfo/trunk/14.0/source/lib/CPUInfo.cpp#2 $

</DEBUG_OUTPUT>
*** Dumping Bread Crumb Stack ***
#** Loading: "/home/lvuser/natinst/bin/startup.rtexe/new_Crio9039 RT 4ch Program.vi"
*** End Dump ***

<DEBUG_OUTPUT>
02/10/18 下午 09时38分15.066秒
DWarnInternal 0x00000000: CPU information collection failed! Assuming minimal functionality.
/builds/penguin/labview/components/CPUInfo/trunk/14.0/source/lib/CPUInfo.cpp(748) : DWarnInternal 0x00000000: CPU information collection failed! Assuming minimal functionality.
$Id: //labview/components/CPUInfo/trunk/14.0/source/lib/CPUInfo.cpp#2 $

</DEBUG_OUTPUT>
*** Dumping Bread Crumb Stack ***
#** Loading: "/home/lvuser/natinst/bin/startup.rtexe/new_Crio9039 RT 4ch Program.vi"
*** End Dump ***

also we find some warning on RT:

lvrt:/builds/penguin/labviewrt/Core/rt_exec/trunk/7.10/os_extensions/
lvalarms_linux/priorityMapper.cpp:255: void {anonymous}::adjustLinPriorities(const tLinThreadList&,int32_t):Assertion `tparams.__sched_priority >=30 && tparams.__sched_priority <=89~ failed

In the attachement i send the files mentioned include the core_dump file. We really need the help.

[gratian.crisan · Accepted solution]

Looking at the attached files I think the thing that causes the restart is the assert in the priority mapper (used by timed structures):

lvalarms_linux/priorityMapper.cpp:255: void {anonymous}::adjustLinPriorities(const tLinThreadList&,int32_t):Assertion `tparams.__sched_priority >=30 && tparams.__sched_priority <=89~ failed

Do you have a lot of explicitly set priorities in your application on timed loops and/or timed sequences? You might be exceeding the number of distinct priorities available. Maybe you can leave some timed loops/sequences at their default priority value, or maybe you can have some share the same priority depending on your application requirements (the scheduler will automatically schedule them).

To give you some context - the priority mapper tries to map the priority number you enter in your timed structures (which if I remember correctly is in the 0-65,535 range) to the actual OS range of real-time priorities which for the two real-time schedulers in Linux (FIFO and RR) goes from 1 to 99. Some of that range is reserved for interrupt threads and time critical VIs. The 30-89 priority range is used by timed structures i.e. roughly 60 distinct priority levels.

[Wang_Liming]

Hello gratian,

Thank you for your suggestion, i will try to assign some timer structures in the same priority. But i find some Real-Time manual say the scheduler will automatically schedule except time critical function (e.g time critical VI and timer structures loop). We will try and find out whether there will be O.K.

Do you have some idea about pam.h functions? When will the RT run su:session close command?  Does some log file record the reboot  event?

2018-02-10T21:38:09.000+08:00 NI-cRIO-9039-01B76F67 su[1984]: pam_unix(su:session): session closed for user lvuser
2018-02-10T21:38:09.000+08:00 NI-cRIO-9039-01B76F67 su[23238]: Successful su for lvuser by admin
2018-02-10T21:38:09.000+08:00 NI-cRIO-9039-01B76F67 su[23238]: + ??? admin:lvuser
2018-02-10T21:38:09.000+08:00 NI-cRIO-9039-01B76F67 su[23238]: pam_unix(su:session): session opened for user lvuser by (uid=0)

[Wang_Liming]

Do you have some idea how to open the Core Dump file?

[Wang_Liming]

Thank you very much, it's the priority problem~~ 

now solved:-)