Data Acquisition Idea Exchange

cancel
Showing results for 
Search instead for 
Did you mean: 
rex1030

Hard Drive cRIO Modules

Status: New

We really need a hard drive crio module for long term monitoring and reliably storing large amounts of data remotely.

 

Hard-Drive-Module-Concept.png

 

Options:

 

1. Solid State Drive: Fast, reliable, and durable. Extremely high data rates. It would be a very high price module but it could be made to handle extreme temperatures and harsh conditions. It should be available in different capacities, varying in price.

 

2. Conventional Hard Drive: This would give any user the ability to store large amounts of storage, in the order of hundreds of Gigabytes. This type should also come in varying storage capacities.

 

For this to be useable:

 

1. It would need to support a file system other than FATxx. The risk of data corruption due to power loss/cycling during recording makes anything that uses this file system completely unreliable and utterly useless for long term monitoring. You can record for two months straight and then something goes wrong and you have nothing but a dead usb drive. So any other file system that is not so susceptible to corruption/damage due to power loss would be fine, reliance, NTFS, etc.

 

2. You should be able to plug in multiple modules and RAID them together for redundancy. This would insure data security and increase the usability of the cRIO for long term remote monitoring in almost any situation. 

 

 

Current cRIO storage issues:

We use NI products primarily in our lab and LabVIEW is awesome. I hope that while being very forward about our issues, we will not upset anyone or turn anyone away from any NI products.  However, attempting to use a cRIO device for long term remote monitoring has brought current storage shortfalls to the forefront and data loss has cost us dearly. These new hard drive modules would solve all the shortfalls of the current storage solutions for the crio. The biggest limitation of the cRIO for long term monitoring at the moment is the fact that it does not support a reliable file system on any external storage. The SD Card module has extremely fast data transfer rates but if power is lost while the SD card is mounted, not only is all the data lost, but the card needs to be physically removed from the device and reformatted with a PC. Even with the best UPS, this module is not suitable for long term monitoring. USB drives have a much slower data transfer rate and are susceptible to the same corruption due to power loss.

 

When we have brought up these issues in the past, the solution offered is to set up a reliable power backup system. It seems that those suggesting this have never tried to use the device with a large application in a situation where they have no physical access to the device, like 500 miles away. Unfortunately, the crio is susceptible to freezing or hanging up and becoming completely unresponsive over the network to a point that it can not be rebooted over the network at all. (Yes even with the setting about halting all processes if TCP becomes unresponsive). We would have to send someone all the way out to the device to hit the reset button or cycle power. Programs freeze, OS' freeze or crash, drivers crash, stuff happens. This should not put the data being stored at risk.

 

I would put money on something like this being already developed by NI. I hope you guys think the module is a good idea, even if you don't agree with all the problems I brought up. I searched around for an idea like this and my apologies if this is a re-post.

 

 

---------------------------------
[will work for kudos]
17 Comments
Broken_Arrow
Active Participant

Neat idea, well explained, but maybe it belongs in the DAQ Idea Exchange?

Richard






TCPlomp
Trusted Enthusiast

Nice idea!

 

Not a solution, but if you need to restart your cRIO when your application halts I would add a physical watchdog. This can be a simple PLC (yes I know that a cRIO should replace a PLC)

Free Code Capture Tool! Version 2.1.3 with comments, web-upload, back-save and snippets!
Nederlandse LabVIEW user groep www.lvug.nl
My LabVIEW Ideas

LabVIEW, programming like it should be!
rex1030
Active Participant

Kudos?

 

We thought the same thing. That's exactly what we did as a workaround but there was a fatal problem. A crio monitoring a crio was just absurd, and we didn't want that. We installed a network controlled power switch we could access over the web. An external physical switch can't perform a soft reset on the device if it's frozen up, it can only cycle power. The crio froze and we cycled power with the remote network power switch like we had done many times before except this time it corrupted the drive. That power loss corrupted 6 weeks of continuous data from a long term test and cost us dearly. It literally turned the usb drive into a brick that won't even mount to a computer as a drive, so no data recovery programs could work. Data recovery services that need clean room work cost upwards of $10,000 and it simply wasn't in the budget. Data lost forever from attempting to reset the device remotely is an obvious fatal problem.

 

The problem our project faces is simply the sheer amount of data we are recording. We tried downloading every day by wireless modem but it was generating 1 Gb of raw binary data daily, which meant download times of around 8 hours a day. We cant record while we download because the crio can't handle it all. We can't stream continuously over the network because the wireless network in the remote area is unreliable which would mean frequent data loss.

 

We need the crio to be able to stand alone and record massive amounts of data reliably. We need reliable and robust storage on the crio.

---------------------------------
[will work for kudos]
Andrew_E
Active Participant

Would the 9802 module already fill option 1? As for the disk drive, I feel that would never pass the pretty tough vibration spec's that we have for c series modules in a compactRIO. However, the best option yet, that would negate both of these is to hook up a USB-Based Storage Media to a controller via USB, like the High Speed USB on the cRIO-902x. Since it is not physically in the cRIO, we don't have to worry about fitting it in a c Series form factor or matching the vibration spec's.

 

 

National Instruments
RIO Embedded Hardware PSE

CompactRIO Developers Guide
PhillipBrooks
Active Participant

No cRIO experience, just interested. The originator made two comments that preclude the 9802. The FAT16 filesystem is required for the 9802, and this is subject to corruption. The other point is that the 9802 supports a max of 4 GB of data, where the OP was looking for hunreds of GBs.

 

What about adding iSCSI client functionality to the cRIO?  An existing Ethernet interface could be used, and the storage device could be local or remote.


Now is the right time to use %^<%Y-%m-%dT%H:%M:%S%3uZ>T
If you don't hate time zones, you're not a real programmer.

"You are what you don't automate"
Inplaceness is synonymous with insidiousness

TCPlomp
Trusted Enthusiast

How does compressing the data work?

You can install the OpenG zip tools on Crio as well.

 

Ton

Free Code Capture Tool! Version 2.1.3 with comments, web-upload, back-save and snippets!
Nederlandse LabVIEW user groep www.lvug.nl
My LabVIEW Ideas

LabVIEW, programming like it should be!
rex1030
Active Participant

Hey guys, thanks for the kudos!

 

Andrew E - Once again the downfall of all those solutions is the FAT file systems. Did you know that if power is lost while the drive is "mounted" by the OS that the data is corrupted and VxWorks cant format it. The only way to recover the SD card is to physically remove it from the device, put it into a PC and format it, then put it back in the crio! Imagine if the cRIO is 500 miles away. They are subject to unrepairable corruption. So the SD card modules and usb drives aren't cutting it because of the file system.

As far as being able to meet the vibration requirements. Couldn't you just make it known that this would lower the vibration handling of the crio? Not everyone needs to strap the crio to a helicopter, some of us just want the crio sitting on a shelf in a traffic box in Nowheresville, Florida 🙂

 

Phillip Brooks - iSCI is a great idea. Awesome solution.

 

TCPlomp - We thought about compressing the data before sending it as well. Unfortunately, after working with the OpenG guys for a week to get the Gzip working on our crio... we found out compression takes longer than actually tranferring the data uncompressed. [time of compression+sending > time of sending uncompressed]

I tried that too, and I had high hopes, but it would take more time to compress+send than to send the uncompressed data file. In general, compression is one of the most processor intensive things a computer can do, which is actually why they use compression for benchmarking. The limitations of the scaled back processor of the crio compared to modern Desktop PCs was never more noticed. As a side note, it opened me up to the potential of OpenG because we were doing something NI didnt support at the time (file compression on crio) and the OpenG guys got it working.

 

 

---------------------------------
[will work for kudos]
ba_0101
Member

Something like this would be great but I would like to see it use an open standard so a COTS harddrive could be used. Maybe something like an eSATA port module. And if we are talking rugged, make the connector for the drive vibration resistant with some sort of strain relief.

Ray.R
Knight of NI

I like the idea of using Solid State drives.  Not crazy about conventional (disk) drives.

 

I also like the idea of using external USB drives.  If so.  Make sure it supports USB-3 which is WAY faster than USB-2.

 

So Solid State Drive or USB-3. 

gsussman
Active Participant

Kudos!

 

I like the idea of an external hard drive adaptor that can be plugged into the slot. If the adaptor supported a standard archetecture such as SATA or eSATA then the end user could select there own drive based on the needs (standard rotating drive for hundreds of gigabytes of storage or SDD for super fast storage).

 

Above all, just like the OP stated, the drive must run a format that is uncorruptable for a power cycle event.