Error -52018 on cRIO 9068 via FPGA write from host

[dest2ko]

I am trying to determine what a particular error (52018) means but have very little context to help understand it. Searching the forums, I see a couple threads with this error pertaining to the PXi system, but none to the cRIO. I'm not sure how the information in these threads applies to my particular case, so I thought I'd ask if anyone's encountered it. My error message does not specifically say "A hardware failure has occurred. The operation could not be completed as specified," but rather just where it occurs: the Invoke Method: FIFO.Write in one of my VIs. This FIFO write has been there functioning for years without issue, but this error is new.

I haven't had much issue running the cRIO RT host VI from my laptop, so when I probe for issues, I never see them. It mainly occurs when I deploy the code / run at startup to the 9068, and whether or not I get it is intermittent.

Attached is a condensed version of my init thread, which runs just prior to my main VI. One change I made recently was one initial pull of data from the FPGA just after I load the bitfile, reset and run it. I have my Run toggled to not wait until done, then immediately make 1 pass through it before it's handed off to the main application cycles. This is just to get an initial value to some other functions prior to starting. I'm not sure how this would pertain to the -52018 error code occurring at the FIFO write, which doesn't even execute here, but thought that maybe something is causing issue if I'm running it and not opting to wait. I will post results here after toggling it to wait.

Any thoughts appreciated.

[dest2ko]

Just noticed what I was saying about the Wait option on Run FPGA method. Obviously I don't want to trigger it to wait. Maybe giving it a 1 second pause between running and reading data will be my next approach.

[Michael_Strain]

What version of RIO do you have installed on the device and what version of LabVIEW did you use to compile the bitfile?

With recent versions of the driver we have started printing information when this error occurs.  After you reproduce the error, could you try checking the serial console for messages or sshing to the target and running "dmesg"?

[dest2ko]

CompactRIO 15.0 - Aug15

NI-RIO IO Scan 15.0

I am compiling everything in LabView 2015 Professional running on a Win7 machine - 15.0f2 32-bit

I have included pictures of everything. Do you think this is a faulty hardware issue? I have tried it across 2 different controllers. These images are from the other day, but it was definitely happening around that time. I will pull more tomorrow morning and post it then along with the dmesg ssh logs. About 9AM CST.

Just to be certain, are the attached images what you meant by the serial console? 

[dest2ko]

Here are the dmesg ssh logs, both during bootups where the incident occurred. 

The symptoms of this error are no I/O function to the cRIO. 

Does anything look peculiar? This is becoming a big concern from a production standpoint. I will be trying module swaps next. 

Thanks for your help.

[dest2ko]

Module swap of the 9476 did nothing.

Up next, I have now removed a timed loop from my RT's FPGA data acquisition thread. This was previously a lower priority timed loop responsible for pulling Modbus serial data from the LV I/O server. It is run in parallel within the RT's FPGA data acquisition thread (a separate, concurrent loop independent of the main FPGA/DMA acquisition loop within the RT's FPGA daq thread.) Will post results shortly.

[Michael_Strain]

Your logs look normal. 

This issue sounds a lot like an issue that was recently found and will be fixed in an upcoming CompactRIO release.  If you talk to support directly, you can reference CAR 680240.  Basically for this issue, some FIFO logic on the FPGA is interpreting something that isn't an error as an error and bubbling it up. 

Ideally we would have you recompile your bitfile with the new version of CompactRIO installed and see if that resolves the issue.  Since its not yet released, I've pointed one of their support people at this thread so they might contact you.

One workaround you might try is to clear the error when it occurs and then Stop and Start the FIFO.  If everything works after that, then it is likely the known issue I described above.

[dest2ko]

Thanks Michael.

As an update, I changed the timed loop to a standard while loop with an equivalent Wait, and everything appears to be working consistently. I have cycled the system about 30 times and run through the application in an appropriate test environment. Nothing's thrown 52018 yet. I will continue to monitor this deployment throughout the day and, if the error remains resolved, I'll provide an additional log file if you think that will be helpful.

If there's anything I can discuss to advance troubleshooting on this, let me know. I don't currently run any code in my FPGA target VI that accesses the serial Modbus data. It's brought in through I/O server, presumably scan engine, or however the PSP process within the SVE that maps the 3rd party slave address abstracts that away. With 16 DMAs, the 9068 seemed to be forgiving enough to not need to worry about utilizing both. I have never gone in and explicitly configured anything pertaining to "hybrid mode" on this system. The chassis programming mode is still declared FPGA from LV.