What is the best method for appending data to a file stored on a usb flash drive to avoid too many write cycles?

[SubmarineEd]

Hi

I am trying to store a large stream sampled data on a cRIO into the USB flash drive.  

I believe there to be two ways to append data to a file:

1. Open the file outside the loop, and then write to the file in a loop, then close the file outside the loop.

Pros ; Faster and more efficient as opening and closing files in a loop is hardware resource heavy.

Cons ; If the program fails part way through, all the data is lost. 

2. Open the file, write to the file and close the file, all in the loop.

With the opposite pros and cons as option 1.

As my cRIO will eventually be deployed in a harsh environment and the data is valuable and processing speed is not a concern in this project, I am leaning towards option 2, but after a few basic calculations I realise that I will be opening/writing/closing the file many thousands of times and I wanted to know if this will wear out the flash memory? 

My uncertainty is due to not understanding what happens when a file is opened written to and closed.  Does it write the new data to new areas of the flash, or does it rewrite all the data over the top?  The latter would cause me problems in the not too long term!

Any help gratefully received.

Many thanks and regards,

Ed

[Sam_Sharp]

Flash memory is normally rated in the order of millions of read/write cycles, rather than thousands.

You didn't mention what file format you are writing the data in but Option 1 doesn't have to result in losing all of your data - if you're just appending to the file you will only lose the new data you're trying to append. There are also various options/VIs to help reduce the data you lose (e.g. disable buffering, 'flush file' VI). I would definitely want to avoid rewriting the entire file to disk each time and unless you're only writing to the file once a minute or so I would definitely open the file outside of the loop.

Also, if your data really is that valuable, I would want to have some sort of backup of the data and/or redundancy in the flash drive (e.g. save the data to multiple drives).

I'm not 100% sure on this, but opening/closing a file reference won't automatically cause it to re-write the entire contents back to disk - the file is broken up into 'sectors' and I'm pretty sure it'll only update the ones it needs to with the new data.

[SubmarineEd]

Thank you Sam

It's a text file and yes, I am looking into having two flash drives with one local and the other remote with identical data.

You're right about not having to close the file to save the data.  I just ran a test with the open and close outside the loop and write inside and hit the abort execution button and the data to date was saved.

As you said the file open/write/close method is unlikely to rewrite anything but the latest segment but I have a concern that there may be a bit at the beginning of the file which says how big it is, and this bit will need to be rewritten every time.  Or the FAT itself will have to be rewritten every time.  I hope I am wrong but if this is the case it won't take too long to wear out the drive if I'm appending the file every second for many days.  

Any ideas?

[Bob_Schor]

Don't know the specific answer to your question (about Flash Drives), but how about a compromise?  What if you combined both Approaches 1 and 2?  Depending on your tolerance for "lost data" vs "Safe Data", you could use Method 1 (open, write-in-loop, close) and every minute, or hour, or day, close and then re-open the file.  For even better safety, you might consider using two file names -- open File A, write to it for an hour, close File A, open File B, copy File A into File B, write File B for an hour, close File B, open File A, copy File B into File A, etc.  If you use a Queue to hold the new records to be appended to the data file, this should give you a sufficient "time buffer" (thanks to LabVIEW's ability for parallel processing) for you to do the Open/Copy operation before adding the new elements present in the Queue.  The drawback of this method, of course, is that you'll have basically two copies of the data on the Flash Drive, hence can only store half as much, but all things considered, this may be a "cheap" and safe solution (data are much more valuable than a $100 flash drive).

Bob Schor

[SubmarineEd]

Thanks Bob

I think I will have to impliment something allong the lines you suggest and at least close the file every now and again.  Although this doesn't help me if the flash drive reaches it's rewrite limit and breaks.

I have also had a read online about how a FAT storage system works and it seems that the FAT itself only stores the location of the beginning of each file, not the length or end position, and therefore will not need to be rewritten when a file gets appended with more data.  Plus the files will have an 'end of file' at the end, so appending data should just rewrite the last segmennt only  - Brilliant!!

I'm still not 100% sure that appending a file doesn't rewrite anything but the last segment, although I appears as if it shouldn't need to, so I'm going to keep this thread open incase anyone else knows the answer for shure, but I'll carry on assuming that it's not going to be a problem until someone tells me otherwise or I have a catistrophic data loss event!  I prey not the latter 🐵

Thanks to all for reading and thanks to Bob and Sam for their help!

Regards

Ed

[Bob_Schor]

You didn't say what type of file it is.  If it is a binary file, you should open it and then use a function to position you to the End of File (this might work with other file types, but my experience has been with binary files).  If you do not do this, you end up overwriting the file!!!

BS

[BuddyHaun]

I had a project where I had the same situation. I used TDMS files, opened the file and only closed it after 24 hours and started a new file. I left buffering enabled in the TDMS file create. I logged several months of data with no problems using this approach. I used a 64 GB flash drive so that I could retain all the files, that made sure that each new file was written to a new area on the flash drive. I did have a routine to clear old files once the drive reached 90% capacity. My application was on a CRIO so I am sure the same approach would work on other systems. If you need to see data that spans more than 24 hours it is easy to write a routine to concatenate TDMS files. Add logic to close the file if you shut down the program prior to the end of 24 hours and you shouldn't have any problems. Each day I would copy the previous day's file to another system but did not have issue FTP'ing partial files off of the system either.

[SubmarineEd]

Hi Bob

It's a text file.  But I think your answer may still apply.  Do you mean the 'Set file position' function set to 'end'?  

So... If I understand it, there should only be a maximum of two writes per segment.  The first, writing EOF at the current EOF and the second over writing EOF with new data when appending the file?

Thank you again

Ed

[SubmarineEd]

Thanks Buddy!

This sounds spookly similar to my project so reassuring to hear that it worked!

I will look into TDMS files.

Thanks again

Ed

[Bob_Schor · Accepted solution]

The "magic" isn't in the File Format (TDMS vs Text vs Binary), but in the "lifetime" of the file.  This solution creates a 24-hour file, and keeps a different file for every day.  If the USB drive "dies", you lose everything that you haven't backed up somewhere.  If the program dies (or errors out), you lose the current day's data.  You end up with multiple files (which can be a Plus or a Minus), but still need to make the trade-off between (time) length of file and number of file opens (which relates to your worry about USB longevity).  Incidentally, I suspect that a good-quality Thumb Drive (that's what you mean, right?  Or do you mean a USB-powered hard drive?  Comments probably apply to both) today can be expected to not die on you just because you do a lot of read/writes (of course, you could get the one-in-a-thousand ...).

Bob Schor