opening large tdms files in excel takes forever, can anything be done?

[labview12110]

After recording some daq channels for 4 hours my file is 120mb large, with about 20columns of data and 150k rows. Opening this in excel takes at least 5min, several not responding screen fades, and growing fear that all my data is impossible to get to.

What can I do about this? Is there a split tdms file option I can use or a way to speed up excel maybe?

[mikeporter]

The solution is to not make the TDMS files so large. Perhaps modify the code so it starts a new file every 1/2 hour or something. Could you post a screen shot of the part of the code that is doing the saving?

Something that just occurred to me is that the TDMS file format is optimized for writing - not reading so stuff is just sort of streamed into the file with an index keeping straight what goes with what. The results is a file where the data from individual channels can be very fragmented, like parts of a file on a hard drive. I think there is a routine for defraging a TDMS file. Alternately, you could save the data in a temporary file (not necessarily TDMS) and resave it to the final TDMS file after all the data is collected.

Mike...

[Hooovahh]

How big is your index file?  It should be very small, say less then 50Kb.  If it is large, or close to the file size of the TDMS file it is a sign of a fragmented file.  Run the Defrag function on the TDMS palette to reduce the size of the data and index file.  This should help with loading times.

[labview12110]

Yes it is quite large actually. Would that be before my flush tdms buffer outside the while loop or just after the last tdms write block I have inside the loop?

[Hooovahh · Accepted solution]

The input to the defrag is the file path.  So this operation takes place after the file has been closed.

EDIT:  Now hopefully the defrag doesn't take 5 minutes.

[mikeporter]

---

@Hooovahh wrote:

EDIT:  Now hopefully the defrag doesn't take 5 minutes.

---

And if it does, you can try my other idea. 

Mike...