LabVIEW Embedded

cancel
Showing results for 
Search instead for 
Did you mean: 

Error 6 writing to USB-stick on cRIO-9068

Hi Guys,

 

I am currently running in to trouble when I stream to my USB-stick on my cRIO-9068. (Linux)

With what seems to be stocaistic timing upon opening the reference to the file I stream to I get an Error 6. 

 

Error 6 occurred at an unidentified location

Possible reason(s):

LabVIEW: Generic file I/O error.
=========================
NI-488: I/O operation aborted.

 

When this has happend I lose any write-priviledge but I am still able to read whatever is on the mapped /U/-drive.
The error can happen everywhere from start-up to several hours into a run. The read-only-priviledge is latched and I cannot get write-access by using the LabVIEW "set permission" primitive(generates and error) and I don't have write-access to the USB via the FTP-server either.

This can only be mitigated by cycling the power to the USB-stick. I.e. taking it out of the socket(and inserting it again) or cycling the power to the cRIO.

 

I have the newest firmware and drivers for NI-RIO, real-time etc.

Modus Operandi:
Every 2 seconds I write to my data file.

WriteAction.png

 


Every 10 seconds I write to my statitics file.
Both write-actions are the same kind.

 

NB! I have stress tested this by allowing them to write very fast and in parallel to test if this is an issue with blockage/arbitrage.

 

 

I have run the cRIO in 2 modes:

1. On a timer that turns it on for 5 minutes and shuts it down for 2 minuts by cycling the power.

2. Permanent power mode where I reboot if any error occurs.

As described earlier the error 6 occurs at seemingly random intervalsSmiley Frustrated

 

Any ideas guys? What is going wrong?

 

Is there a workaround? To cycle the power of the usb-stick programatically, maybe using the "sync"-command or "umount/mount"?

If I can catch the error and revert the usb-stick to write-access again, that will be great! as the write timing is not critical, I can easily buffer the data and wait till the USB-stick has been powercycled?

 

 

Bjarke Dahl-Madsen - CLA,CLED
0 Kudos
Message 1 of 33
(9,375 Views)

Something that would be beneficial would be to attempt to look at the file from outside of LabVIEW. Enable the console (or the sshd server) and attempt to access the file (e.g. run the command "cat /U/filename.ext") and/or check the kernel message for error messages (i.e. run the command "dmesg | grep -E "sd[a-z]"", this will output the kernel message ring buffer, filtering the messages looking for storage-related messages).

 

Another thing to check is to see if the issue reproduces with another USB storage device (it could be that you're seeing some hardware problem)

Message 2 of 33
(9,363 Views)

Thanks Brad will try that tomorrow.
You seem to know your way around Linux commands Smiley Happy

Is there anyway to cycle the power of the USB-stick (eject/insert) programmatically?

Bjarke Dahl-Madsen - CLA,CLED
0 Kudos
Message 3 of 33
(9,358 Views)

The approach that I would probably take would be to tickle the sysfs entry for the driver, unbinding the device from the driver and rebinding. The issue you'll likely run into is that the LabVIEW RT process is run as a slightly-above-average-priviledged user who will likely be unable to tickle the sysfs entries (/sys/bus/usb/drivers/usb/unbind and bind, writing into the file the device you wish to unbind, e.g. "1-1"), in which case you could do something to eitehr provide a means to talk to a process running as root (a daemon started on boot, e.g.) or change the LabVIEW process to run as root (which is a really, really bad idea, for security reasons, if you care about that).

 

That said, I would think that the better solution would be to get to the bottom of what you're seeing in lieu of breaking out the bailing wire and duct tape

Message 4 of 33
(9,351 Views)

Hi BradM.

 

 

I have been working with Bjarke on this issue. I have been trying to reproduce the issue by making a simple VI, that write to two files in parallel. This means, that in each iteration, I'm creating two new files, and they will be closed as well in each iteration. The loop is iterating every 10s. I have been running this overnight, and thereby generated almost 12.000 files in the same folder on the USB stick, to really stress test the RIO and the USB stick.

 

RIO Write.png

 


I have not been able to reproduce the issue on my end. Have you heard of anyone else that are having this issue??

 

Bjarke is seeing this issue at very random intervals, so we would like to try any workarounds, where we do "something" if error 6 occurs, just like Bjarke is mentioning. However, most of the things require that lvuser has root access. We know that this will compromise the security, but would like to try it out. Do you know how we get root access from within LabVIEW?

 

Thanks for helping!

 

Best Regards

Alex E. Munkhaus
Certified LabVIEW Developer (CLD)
System Engineer
0 Kudos
Message 5 of 33
(9,337 Views)

I really feel that this is a hardware problem. Simply changing the application to account for hardware issues is the recipe for delaying the inevitable total hardware failure. Stated another way, I would much see the same failure mode with other USB storage before resorting to changing LabVIEW to run as root. Additionally, this could impact files that need to be accessed by other NI software that is created by LabVIEW (as it now is owned by root)

 

I'm not going to recommend the sloppy fix since, even with all of the testing that is done (including hardware validation and software performance testing), unique customer applications that stress the USB system, not to mention the horrible, horrible abuse that R&D give these controlllers as we develop, no reports of this problem have come up.

0 Kudos
Message 6 of 33
(9,319 Views)

Thx guys for working on a solution.

Status is currently that I am pulling my hairs out  trying to recreate the error.  I have not seen it for 24 hours of continuous testing. Even though it is the same same executable(and USB stick) that has been running for 6 days.Smiley Embarassed


I was hoping to be able to post an error message based on your feedback Brad. I will let the system run overnight (again) and see if anything happens.

Thank you also for your patience 🙂 

Bjarke Dahl-Madsen - CLA,CLED
0 Kudos
Message 7 of 33
(9,315 Views)

After 48 hours of no failure I copied the image to a duplicate system.
Same .exe etc. but new cRIO and USB-stick. - This in order to eliminate issues with hardware failure.

 

Finally the error started to show up.

I coerced it into failure mode and I currently have an open putty connection to the ssdh.

the cat- command return the content of the files with no problem.

 

the dmesg command returns this:

 

Screenshot 2014-05-22 16.34.12.png

 

And indeed it seems that there is an error... and it is not all inside my head! *phew*!

The USB stick is now read-only.
What to do?

Bjarke Dahl-Madsen - CLA,CLED
0 Kudos
Message 8 of 33
(9,291 Views)

How are the drives that you're using being formatted before use in these systems? It seems that there is something awry with internal structure of the filesystem, and most reports I've seen online with this sort of behavior seem to be resolved with simply unmounting the filesystem and running fsck on the device. You may want to try the following from the terminal:

 

//stop the device from being used

umount /media/sda1

 

//have some system tools clean the FAT internal structures

//make sure to let us know what the tool reports

fsck.vfat /dev/sda1

 

//reenable the device

mount /media/sda1

 

//check that things are still accessible

ls /u/

 

If things seem to work, try reproducing the issue after cleaning up the filesystem. And make sure to let us know how you are creating the filesystem as it is being used initially.

0 Kudos
Message 9 of 33
(9,284 Views)

The drives are formattet by using FAT32 and a quick format using the native windows tool.

 

//stop the device from being used

umount /media/sda1

Done... 

 

//have some system tools clean the FAT internal structures

//make sure to let us know what the tool reports

fsck.vfat /dev/sda1

Upon trying this:

-bash: fsck.vfat: command not found

 

//reenable the device

mount /media/sda1

Upon trying this:

mount: can't find /media/sda1 in /etc/fstab

 

//check that things are still accessible

ls /u/

 

I am sorry I am such a neophyte at Linux -  your help is extremely appriciated!
Did I misunderstand your instructions?

 

Bjarke Dahl-Madsen - CLA,CLED
0 Kudos
Message 10 of 33
(9,276 Views)