Help needed!! Why do I keep losing the connection to my cRIO?

[tdog]

Hi All,

I've got a real time system with the following setup.

I have got a cRIO out in the field running a real time application that accquires data from various thermocouples and sends the data to the my host computer over a wireless radio link. I am using a wireless ethernet modem with a data transfer rate of 100 kb/sec. The modem is being shared by 12 other radios also transmitting data to the host PC.

So, basically, I've got the host PC on one end and 13 field computers (one of which is my cRIO running a real-time application) on the other end. The 13 field computers are sending data to the host PC over wireless radio links utilizing a single wireless ethernet modem. The data from the 13 radios is placed on a que and the modem cycles through the que enabling the data from each radio to pass through one at a time.

My real time application is sampling the thermocouple data every 1 sec and sending it across to the host application via a shared variable

The host PC is also running an application that acts as the user interface.

My problem is this:

My system runs OK for a couple of minutes and then suddenly I lose the conection to the cRIO. Once this happens, I have to manually reset the cRIO (i.e. cycle the power back on) to re-establish the connection. This is both tiresome and inconvenient as the system needs to be able to run continuously for several hrs.

What could be the possible causes for this loss in connection? Is it possible that a clash between signals causes the modem to drop the connection to the cRIO? If so, is there a way to monitor the connection between the host and cRIO and automatically reset the cRIO in the event of a lost connection?

Any help would be highly appreciated.

Thanks,

Taizoon

[GregFreeman]

I am currently having the same problem. I'm hoping it fixes itself.

[nathand]

How are you communicating with the cRIO?  Is it the LabVIEW development environment that loses the connection, or shared variables, or your own TCP protocol, or something else?

Have you checked the crash/error log on the cRIO?  You can do this through the project, by right-clicking on the cRIO target.  I don't have access to a cRIO right now but if I'm not mistaken there's a User LED that you can set in your program.  You could put in some code that turns it on and off regularly, then see if it's still blinking when the connection dies - this will help determine if your entire program fails, or just your network connection.  Most often connection problems occur when something else on the cRIO running at higher priority takes up too much CPU time, preventing the networking code from responding.

[Ben]

---

@nathand wrote:

...  Most often connection problems occur when something else on the cRIO running at higher priority takes up too much CPU time, preventing the networking code from responding.

---

Or it ran out of memory...

or the watchdog timed out

Ben

[GregFreeman]

Thanks for the suggestions. I think I was overflowing my FIFO so it may be a memory issue. However, it dies hard when I place a probe on my FIFO read data. Since this seems to be repeatable, NI may be able to look into it.

Anyways, I'll try some of my own troubleshooting and your suggestions then if I'm still having problems I'll be sure to check back.

[tdog]

Thanks for all your suggestions. I'll take them into considerations and see what happens.

Cheers.

[Lisa7603]

I am currently connecting to an RT cRIO 9024 and I am having the same problem. I noticed a few days ago before this started happening that the CPU of the Rio was gettting to 90-95%. We deleted some vi's and now the CPU is lower, 30-40%. Sometimes the code runs fine, but sometimes when we make changes and run the code asgain, the following error would come up "Failed to connect to target". At times, the code starts running fine and the PC suddenly looses connection to the Rio and the following error shows up " Lost connection to real time target" even though the power is still on. Also, when running the code on the PC, labview would fail to deploy the variables or shows up the following message for a few seconds"waiting for real time target(RT CompactRio) to respond" and continues with the deployment or just stops the deployment.

If anyone has a solution to this problem, please respond. I really hope there is a fix to this. 

[Chris V]

Hi Lisa7603,

Most often this occurs when a task running on your cRIO is taking up all of your processing power and not allowing the communication task time to run. It was a good idea to try to reduce the CPU load, but you should really consider taking a look at scheduling on your whole system to make sure that all of the tasks required are allowed to run.

The Real-Time Execution Trace Toolkit is very helpful for analyzing your thread usage.

Have a great day,

Chris V

[Nemesis]

I have similar problem. I am loosing a connection while I am running vi. Everything is fine but suddenly my FIFO buffer is full and everything freeze. If I remember well I did not have this kind of problem with cRIO 3.3.0 but as I got new Labview 2010 I use now 3.6.1 drivers and the problem appeared. I have NI 9024 plus NI 9112. I found something interesting. In my vi I do open/write/close on txt files. I wanted to check if the problem is caused maybe by 4GB internal disc. I changed saving files to USB key (from c to u letter).I did not changed anything else. Since then I do not have any problem and everything is working well but I cannot use internal memory to read/write. I cannot explain it and unfortunately solve.

Kamil

[Chris V]

Hi Nemesis,

Are you attempting to do your file writing inside of a timed loop?  Or is it just inside of a normal while loop?

Have a great day,

Chris V