Flushing TDMS file without creating delay

[LarsAlmås]

I have a problem where my datalogger has to run for extended periods of time (days).

To prevent loss of data in case of computer crash, I would like to flush the data into to TDMS once every 15 min.

The problem i have is that every thing I have tried so far results in a increasing delay between real time and the latest sample (the logger lags behind).

After 3 days of logging its about 10 mins deviation.

The file takes a starttime, and number of samples and multiplies it with dealy between samples to get supposed time and date.

I'm assuming it skips a few samples every time it saves the file and therefore it won't have enough samples.

So far I've tried closing and opening a new reference, flushing, flushing in a seperate vi parallell to the main vi.

If anyone has any experience in this I would be glad to know if there are any tricks to this.

[Hooovahh · Accepted solution]

So what I'd recommend to you is to do as you are, defragging periodically, until the file size, or defrag time increases to some specified amount which will likely vary for you.  Like for instance lets say you keep defragging the file, but once the file is 24 hours old, make a new file.

Then after another 24 hours make a new file, and continue this until the test is done.  Then you will have one file for every 24 hours, and you can then use the TDMS merge functionality to combine them.  I presented on TDMS at NI Week 2016, but also updated the presentation and shared it here.  Slide 13 talks about a buffering technique where new files are created, then sent away for processing until all files are done which might be useful to look into.

[charlieallanp]

I get a page error when I try to open your presentation

[billko]

Are you using a producer/consumer design pattern?

[cbutcher]

It sounds to me like you're running the acquisition and logging steps in the same loop.

I'd suggest the following:

• Upload your code so we can make better/more targeted recommendations
• If you are logging in the same loop as acquiring, use instead the Producer/Consumer design suggested by Bill

By separating the acquiring from logging steps, you won't have delays in the latter cause problems for the former.

You can also consider trying to get a "real" timestamp from the acquisition step, possibly by using Tick Count or Get Date/Time in Seconds, depending on accuracy requirements. This might improve the options you have to satisfy your goals (maybe, depending on your goals 😉 )

[Hooovahh]

---

@charlieallanp wrote:

I get a page error when I try to open your presentation

---

That is a real shame.  It seems like the Rocket City LabVIEW user group has disappeared.  I can't remember if they recorded it or if I only had my slides.  Attached are my slides from that presentation, along with the demos I showed.  Probably 80% of it is the same as the NI Week presentation.

[LarsAlmås]

We went with a solution along the lines of Hooovahh's suggestion back when this post was created. 
Going to mark this as the answer as this was solved some years ago and I had completely forgotten about the post. 

[Hooovahh]

Oh and one other thing I forgot to add.  Since making that post I released the Tremendous TDMS Toolkit on VIPM.IO.  Here is a youtube video demonstrating it.  It has several functions like new file size and frequency, among other things..

[charlieallanp]

Thanks, I found the site (VIPM) but neither the download or install buttons work.