TDMS file backup

[nileshborhade]

Hello, 

         I have to design a data acquisition application in which acquired data is stored in tdms file. I am supposed to handle the issue of sudden power failure. Can anybody Please guide me on how to backup the open TDMS file. Because, in a deliberate attempt of testing it, I saw that once the running tdms file is corrupt, it can neither be repaired nor opened. I am going to design a stand alone labview exe application. Therefore, I want to design a VI which will take care of the data of the open tdms file. Can I use Create System Image VI to create the system image, may be every 1 second during the execution? Or is there any better solution?

Thank You,

Nilesh Borhade 

[nileshborhade]

Hello,

        Sorry, If I have not clearly mentioned the problem, I put it again. I want to design a stand alone data acquisition application in Labview where the application runs without any user handling i.e. self running 24x7. I am going to use TDMS file format to store acqired data. So, in case of power failure my tdms file being executed for writting the data, will corrupt and I will not be able to recover the data programmatically after a restart. I am planning to have a backup operation simulatenously running with data writting. How can I implement it? 

Thank You !!!

[josborne]

Great question!  Kudoes from me.  

I ran into this exact problem a while ago.  I tested it by killing my application while it was acquiring data and discovered that the TDMS file was corrupt and worthless.  My only solution was split the storage into multiple files.  So I create a new file when the existing one reaches 500 MB ... so in the event of a crash, I only lose the most recent 500 MB.  This was not ideal because 500 MB is a lot of data!  And if I reduce that size, I end up with too many files.

Here's a clever idea:

Use two TDMS files.  But only have ONE of them open at a time.  The files will essentially be duplicates of each other.  And every X seconds you can alternately write to a different TDMS.  So at any given time, a crash could only potentially ruin one of the TDMS files.  

You will need to maintain an internal memory buffer.  And you'll need to continuously open and close the files, which slow it down.  But depending on how much data you are acquiring, it would work.  It'd function like this:

After 10 seconds, write the previous 20 seconds to FileA.  

After 10 seconds, write the previous 20 seconds to FileB. 

After 10 seconds, write the previous 20 seconds to FileA.

After 10 seconds, write the previous 20 seconds to FileB. 

[josborne]

Or I guess you could just close the existing file regularly and make a copy of it.

[nileshborhade]

Thank You very much for your reply....!!!

I was anyway planning to create short size files for the data prevention purpose but, as you said, loosing 500MB file is a big deal. So, I would like to implement the idea of having two duplcate files but again, then the runtime data which is been updated at a huge rate may overflow the buffer. Also, doesn't this method cause performance problem? Because, I am told not to degrade the performance for other reasons to a considerable extent. May be a trade-off is possible.

I respect and appreciate your second suggestion also. I was also thinking of closing the tdms file every iteration but still the sudden power failure affects the file closing every cycle also.

Even if so, I will implement your idea of 2 files and will test it's impact on performance. Thanks once again...!!!!

Still if you or anybody can give another suggestion, will be thankful...!!!

[DFGray]

To make things faster, TDMS buffers data and only writes when the buffer is filled.  You can override this behavior by either turning it off when you initialize the file or by explicitly flushing the data every time you write.  However, even this will not prevent possible corruption on power loss, since the power loss can occur any time during the write process, resulting in a corrupt file.  The data in the file is not lost, however.  The TDMS file format is documented here.  Try trimming the last, incomplete, segment and deleting the index file, if any.  You can do this with the code posted here.  You may want to put a suggestion in the LabVIEW idea exchange for a TDMS file cleaning utility.