Create a tdms defrag progress bar

[labview12110]

I am saving large amounts of data (32 analog votlage channels from 5 usb daqs all plugged in to the same computer ). Continous samples at 100Samples per second for 1hr or more. The tdms save file is huge by the end of the program so when I hit stop the entire thing hangs while labview runs the final tdms defrag block. I don't necessarily mind this, but it would give me so much piece of mind if I had some sort of defrag in progress progress bar that popped up when this happened. Is there a way to do this? How do I find the current VI state and wire that to a progress bar indicator?

[Hooovahh]

I totally agree, and actually spent a good amount of time trying to split a TDMS file into chucks, then defrag each chunck, then combine them.  The defrag each chunk, and combine each chunk was trivial, but the splitting of a TDMS file, without having to perform the TDMS Open function (which takes a long time for large files) was too difficult and I gave up.

Here are a few suggestions.  Defrag periodically when your program is running.  Spin off a parallel loop or Actor that defrags every 10 minutes or so.  Sure the time needed to defrag will go up each 10 minutes, but the final defrag will be much shorter because much of the file is already defragged.

Or you can choose to make new TDMS files ever so often.  So again say every 10 minutes make a new file, and close and defrag the old one.  Then at the end of the test combine the files using the copy /b system exec command which is much faster than opening each file. 

[labview12110]

That is certainly unfortunate. Is tehre a way to get the current vi status and time to completion for  a subvi step? I would just like some indictor that the program has not crashed and is still running. Once the defrag is complete the file is only 10mb down from 100-120mb so if i have to only wait an extra minute to defrag at the end without all the hassle of splitting and combining and splitting and combining.

[Hooovahh]

There is no current VI status, but if your file is that large there is likely optomizations you can do in how you write the TDMS file, calling the TDMS Write function less often, by combining consecutive writes of the same data type.  Flushing one sample or one channel at a time will cause the header information to need to be written, which is why the file is so large.

So while there is no single solution, you have about 5 different things to improve performance.

Write multiple samples if you aren't already

Write multiple channels and multiple samples if you aren't already

Flush to disk less often

Defrag periodcially in a parallel loop

Create new log files after so much time and combine them at the end

[labview12110]

I tried to incorporate a periodic file defrag but am running into an error when I try and contiune data collection afterwards (error -2501 @ TDMS write). Attached is a chunk of my vi that is for one daq, picture this with 5 daqs and 5 tdms files writing simultaneously.  Where have I gone wrong in incorporating this? My thought behind it is that either X amount of time has passed and it is good idea to defrag and shrink  the file size, or i have a lull in data collection say while changing a setup and can hit the button at that time. I assume that while it is doing the defrag i will lose that amount of data.

[labview12110 · Accepted solution]

Actually I think i just figure it out, if I replace the file path and error wires with shift registers, it passes the correct path to the beginning of the loop again! is this correct?

[B_Sherm]

As far as I can tell that should work. Once you close the reference and then open it back up, it is a good idea to send the new refence to the beginning of the loop.