[TDMS] Write fixed file Size with the FIFO model.

[FM82]

Hi everyone,

I need to create a tdms file, from a continuous acquisition, with a fixed file size. I explain it better; suppose I only want a maximum of 1 GB for the tdms file, if the file has exceeded this size I want to delete the first element inserted and only then add the new data. Similar to a lossy FIFO queue, if the queue size has exceeded the limit, the oldest data will be deleted. I hope I explained myself. Any suggestions would be really appreciated.

Have a nice day,

Francesco

[GerdW]

Hi Francesco,

there is a "TDMS Delete Data" function, which allows deletion of channels in a group.

So you could delete data group-wise…

Technically it would be possible to implement your FIFO-style TDMS file.

BUT: this (most probably) leads to a lot of fragmentation in the file, and you need to clean up the file later on. It may also not reduce filesize immediatly, just after cleanup.

So you will not gain much.

It will be much easier to start a new TDMS file when you reach a filesize limit! Maybe you could implement 10 files, each for 100MB of data: Overwrite the oldest file of those 10 whenever your 100MB limit is reached…

[DGrayEOS]

As Gerd said, TDMS is a poor choice for a circular disk buffer. Unfortunately, TDMS uses a lot of under-the-hood optimizations to avoid data copies that normal users of LabVIEW do not have access to, particularly when streaming from NI-DAQ devices, so any user created disk circular buffer will probably run a bit slower. The key is "a bit." You do have two other decent options:

1. Raw binary - do it yourself with a raw binary file. Be prepared to tweak a lot if you do this, but it will probably give you best performance. In particular, pay attention to the buffer size you write to disk. Around 65,000 points was optimum last time I checked.
2. Use HDF5 - NI does not natively support this, but there are multiple implementations available on the web. An HDF5 file with a fixed sized dataset makes a circular disk buffer easy. You will need to pay attention to buffer sizes again.

[FM82]

Thank you guys. I will think about what of the above ideas (more files or HDF5) works better for me. Yours was a really precious help.