TDMS defragment with display progress (built-in example) question

[anthony797]

Hi,

After my data collection, I got a big TDMS file(260 MB) with 1 group 4 channels data along with big index file(2XX MB). It takes very long time to open it by either MATLAB or Python. I believe that is the well-known fragment problem.

I try to use the built-in example to defrag it as attached vi. What I modify is to set my base path and my file name in the beginning. However, it turned out to be a weird file with 1 different group and lots of channels.

I am wondering how I can do this correctly.

Thanks

[Hooovahh]

Look at the example.  It performs a Create or Replace on the file path you select, it then creates or replaces what is there with a fragmented file, it then defrags the file, and monitors its progress.

If this isn't what you want you need to edit the code to make it do what you want.  I suspect that means you want to just remove all the code that creates and replaces an existing file, and instead just give it to the path to your already created file.

A better solution, but one that takes more work is to not create fragmented files in the first place.  Look at how you are performing the write operations and try to perform the write operation less.  This might be combining multiple writes into a single call, or writing more samples at once.  Most often this problem comes about when a developer chooses to write single point data to a TDMS file.

[anthony797]

Hi  Hooovahh,

Thank you very much. You are right and that works.

I would like to try not to create fragmented file in the first place.

What I have done is shown in attached picture. My whole file is complicated but I believe what matters is the way I save as TDMS. I just used ''write to measurement file'' to steam my collected data (time, temp, stressX, stressY) to TDMS.

Do you have any idea how to avoid creating fragmented data?

Thanks,

[Kevin_Price]

Additionally, you can also reduce your write-time fragmentation by specifying the special property NI_MinimumBufferSize.  See this article for more info.

-Kevin P

[Hooovahh · Accepted solution]

---

@anthony797 wrote:

Do you have any idea how to avoid creating fragmented data?

---

Yes as I stated use the write function less by combining multiple writes into one, or buffering data to call the write less often.  Your Express VIs are useful until they aren't, and then they are almost worthless.  If an express VI works for you that is great, but know that as soon as you want to deviate from its functionality at all you are going to have have to start over with that set of code.

Look into the TDMS primitive functions on the File IO >> TDMS palette.  There are shipped examples in Help >> Find Examples for writing files.  The suggestion of minimum buffer size is a good one, but I've personally not seen it impact my fragmented-ness as much as refactoring the code to write less.