Real-Time process keeps encountering unexpected error and restarting automatically

[DrakeP.]

We're currently testing our robot, and have had the code abort on us about half a dozen times in the last few hours with the same error every time. It will say that we lost connection while waiting for the target to respond, and when we deploy it again, it will say:

"Errors were detected in the target log when connecting to the target:

LabVIEW: (Hex 0x661) The LabVIEW Real-Time process encountered an unexpected error and restarted automatically. Select "Apply" to ignore these errors and continue with deployment"

We have had this issue over radio and over USB. We use the CTRE libraries and have NavX MXP libraries in the project (although we've still encountered the issue when we use a diagram disable to disable all navX VIs). If anyone could help, that would be deeply appreciated. I can provide more information if needed.

[oscarfonseca]

Hello DrakeP.

I understand this is happening with your current robot code. Does it happen too with a fresh LabVIEW roboRIO project? I think the first step is to discard that it's the controller is working as expected (and the issue is not there).

Moreover, what happens if you try the project with a second roboRIO? Do you see the same behavior?

All the best,

[aeastet]

You can start adding a disable diagram around parts of your code until you isolate what is causing the problems. I would look at different things that might be causing the issues like motors, encoder, etc.. You might put a disable diagram around your timed tasks vi and see if that makes the error go away. we need to isolate where is error is originating from.

Hopefully that will point you in a direction.

[DrakeP.]

We did not have the issue happen in the hour or so that we let a new project run. We have had the error on 3 different roboRIOs, but I'm pretty sure (although my memory is kinda fuzzy) we've only ever encountered the error on one of our several laptops (coincidentally the one we use most so that's why we've run into the error several times). Of course, it's hard to pinpoint exactly the problem without several hours testing, and that's tough to do with drive practice using the robot a lot. 

We had a hunch that the issue was a bug resolved in the FRCUpdateSuite 2019.2 mandatory update, but we have since had the bug happen 3 times a few nights ago. Is that computer the culprit, or is it a coincidence and confirmation bias leading us astray and we need to look somewhere else?

[aeastet]

Is there any way you can post your code here or post some pictures of your begin, timed tasks and teleop? We can try to take a look at what you are doing and see if there is a reason to expect the code or the roborio.

[DrakeP.]

Here's a github link for the project: https://github.com/DrakeProvost/2019-Deep-Space-Code-WIP-

The hatch and HAB mechanism VIs in periodic tasks are disabled because we were trying to find the cause of the “unexpected error” issue, and they do not seem to be the cause of it because we were still getting many crashes with those disabled. We didn't try disabling the drive loop because we were using the drive train at the time to test other code. Let me know if you need additional information.

[oscarfonseca]

Hello Drake.P

Thanks for sharing your code. I can see there how everything is in Periodic Tasks but an encoder reset that seems to happen each 20ms of Teleop.vi

Besides of it being a large block diagram, in a first glimpse I don't see an obvious reason why it would crash. I am intrigued however on the amount of RAM memory that it could be using, since we are dealing with some arrays that are re-written over and over.

You mention this behavior happens about a dozen times an hour. Does it always happen at the same time after the code starts running?

Moreover, if you open the DS Log, do you see a high CPU usage? How about battery?

Finally, disabling the drive and letting the roboRIO run for a while would be a good troubleshooting step to understand if this has to do with that code. I understand the drive team has to use it constantly, but perhaps during a meal or sometime when they're not practicing.

All the best,

[DrakeP. · Accepted solution]

I consulted with one of our coding mentors from another team on several occasions about the issue, and our current best hypothesis is bandwidth issues. Even though we seem to only be using 2.6 Mbps (according to the dashboard), his guess is that the large amount of data being transferred when we are soft deployed (by clicking the run arrow in robot main) combined with the 2 cameras we're using is causing delays in the communications that are just long enough for the robot to temporarily disable and think it lost comms. Since hard deploying using less bandwidth we've been trying to use that as often as possible, and out of the (somewhat small) sample size of 6 or so hours hard deployed, we have not encountered the issue. Not a perfect solution by any means, but at the very least it seems like it won't be an issue on the field, so at least there's that. 

Edit: oh, and I forgot to mention. watching the "CPU %" bar on the driver station seems to show that CPU usage is not the cause since it's always pretty low even when we have the crashes. 

[oscarfonseca]

That makes sense. That's the reason I asked how much time had elapsed when it happened.

I've seen similar behavior with my team when we use the interactive execution (clicking Run in Robot Main.vi). The fact it has to update everything back in the computer can overload the network connection and lose connection with the target eventually.

All the best,