Hi all. I could use some advice on designing a disk buffer:
I am gathering lots of data at high speed. I will be storing the data in a SQL Server database. But given that databases and network connections are prone to be slow and occasionally disconnect, I want to first dump all my data to disk. So if there is an error/disconnect/power-loss, I don't lose any data.
Question 1: What is your reccomendation for architecting a disk buffer like this? I tried using TDMS because I like being able to do simultaneous read/writes, but the file gets unnecessarily large and complex. I disabled TDMS buffering to avoid data loss in the event of a crash ... this caused my TDMS files to get huge very quickly. I am considering just using an ASCII file... and maybe keeping the file open and storing the file reference in a functional global so that I can read/write to the file without having to open/close the file all the time. Any tips?
Question 2: What if there is a crash? Is there a potential for the file to become corrupted if the file reference remains open? Is ASCII or TDMS a better solution with regards to the potential for file corruption?
Thanks for your advice!
http://www.medicollector.com