Advice on disk buffering / Huge TDMS files

[josborne]

Hi all.  I could use some advice on designing a disk buffer:

I am gathering lots of data at high speed.  I will be storing the data in a SQL Server database.  But given that databases and network connections are prone to be slow and occasionally disconnect, I want to first dump all my data to disk.  So if there is an error/disconnect/power-loss, I don't lose any data.  

Question 1:  What is your reccomendation for architecting a disk buffer like this?  I tried using TDMS because I like being able to do simultaneous read/writes, but the file gets unnecessarily large and complex.  I disabled TDMS buffering to avoid data loss in the event of a crash ... this caused my TDMS files to get huge very quickly.  I am considering just using an ASCII file... and maybe keeping the file open and storing the file reference in a functional global so that I can read/write to the file without having to open/close the file all the time.  Any tips?

Question 2:  What if there is a crash?  Is there a potential for the file to become corrupted if the file reference remains open?  Is ASCII or TDMS a better solution with regards to the potential for file corruption?

Thanks for your advice!

[josborne]

Here is my example code (as a snippet and also as attached VI).

My TDMS file gets huge very quickly.  Any ideas on how to reduce this size? 

[josborne]

Sorry.  Think I will answer my own question.

The answer is here:

http://digital.ni.com/public.nsf/allkb/63DA22F92660C7308625744700781A8D 

And the above file I uploaded actually does not get so unnecessarily lagre.

Case closed.