What could be the best file format to save data for my application?

[charmelony_]

Hi crosslulz,

The reason of open-write-close in each loop was to make sure data are saved to disk. On the other hand, without the close file function, the VI writes data to disk. I even killed the working VI by Windows Task Manager, the data were still saved. As far as know, close file function is being used to transfer the written data from memory to disk. Do you have any input about that? It seems that windows handles some different way to write the data.

[LLindenbauer]

---

@JÞB wrote:

---

---

Excel is a proprietary file format. 

---

I would argue that this is no longer the case, due to the x - formats being based on the Office Open XML standard: https://docs.microsoft.com/en-us/openspecs/office_standards/ms-xlsx/f780b2d6-8252-4074-9fe3-5d7bc483...

In my experiments, I measure different type of outputs with respect to given inputs. In the first figure, you can have some idea about the data output. This is only for one channel of the rig, I have up to 20 channels by using multiple test rigs. In the second figure, block diagram of save VI can be seen. Data are saved to normal file with intervals that I can decide. Experiments will continue couple of months -maybe even a year(s)-. I was updating the VI scripts and could not decide whether I should stick the current saving format, or a different one e.g. tdms. All data will be further processed with respective softwares. The VI has an automatic control function as well. I am going to have lots of long data files. I do not want to lose any data point (only related with saving) and I need to access to the data file while the VI is working (currently I use notepad++ for that). Do you suggest me to switch tdms format?

What kind of data processing will happen downstream? It might make sense to start from there to keep the interfaces simple.

What kind of machine will do the logging? I would be hesitant to have all data of an extensive long-running experiment being a "normal file on the rig".

Do you suspect that the data format might change during the experiment? I found rigid (tabular) data formats to become incredibly cumbersome when you have to patch on more data.

You might want to check if HDF(5) or even a dedicated database (SQLite) on a different machine could be right for you.

[Kevin_Price]

Windows will generally maintain a memory cache for data you (try to) write to disk.  The data initially goes into the memory cache and then Windows makes up its own mind about when it actually commits it to the physical disk.  Often that happens very quickly, but not always.

My understanding is that calling a Close() or Flush() function will work synchronously -- i.e., the function call won't return until the cached data has all been committed to disk.  That *can* slow down your file operations so it's usually helpful *not* to do it all the time.

You might consider an approach I've taken in the past.  First, the evaluation.  The reason to flush the disk cache is to make absolutely sure that in case of a power failure or blue-screen crash, I still have a record of my data up until that point.  So the next question is, if there was such a catastropic failure of the test system, what's the true consequence of missing that final second before the crash?  The final 10?  The final minute?   Etc.

    Often, the practical answer is that the crash pretty much ruins the test anyway, and it's pretty unlikely that something you may have missed during the final 150 msec would have changed that assessment.   So I would often set up my logging loop to do simple writes  and only call Flush() once in a while, maybe in the range of 5 sec to 1 minute.

-Kevin P

[crossrulz]

---

@charmelony_ wrote:

As far as know, close file function is being used to transfer the written data from memory to disk. Do you have any input about that? It seems that windows handles some different way to write the data.

---

When you write to disk, then the data goes into a cache.  Normally, the OS will perform the actual write every so often or when there is enough data in the buffer to write to the disk.  I see the data being saved to disk as I log data.  As Kevin already stated, you can use Flush File or TDMS Flush to force the OS to write the file cache to disk.  There is no need to actually close the file.  Constantly opening and closing the file is really expensive, so I would avoid that if possible.

[charmelony_]

---

@Kevin_Price wrote:

Windows will generally maintain a memory cache for data you (try to) write to disk.  The data initially goes into the memory cache and then Windows makes up its own mind about when it actually commits it to the physical disk.  Often that happens very quickly, but not always.

My understanding is that calling a Close() or Flush() function will work synchronously -- i.e., the function call won't return until the cached data has all been committed to disk.  That *can* slow down your file operations so it's usually helpful *not* to do it all the time.

You might consider an approach I've taken in the past.  First, the evaluation.  The reason to flush the disk cache is to make absolutely sure that in case of a power failure or blue-screen crash, I still have a record of my data up until that point.  So the next question is, if there was such a catastropic failure of the test system, what's the true consequence of missing that final second before the crash?  The final 10?  The final minute?   Etc.

    Often, the practical answer is that the crash pretty much ruins the test anyway, and it's pretty unlikely that something you may have missed during the final 150 msec would have changed that assessment.   So I would often set up my logging loop to do simple writes  and only call Flush() once in a while, maybe in the range of 5 sec to 1 minute.

-Kevin P

---

I have different experimental sequences. Save interval time is either 30 seconds or one minute (data at that point). The last data point is not important in case of failure. On the other hand, I absolutely do not want file corruption due to system crash. What I understand from the replies is that I do not need to open-write-close in each loop. Do i still need to flush the data?

[LLindenbauer]

---

@charmelony_ wrote:

---

On the other hand, I absolutely do not want file corruption due to system crash. What I understand from the replies is that I do not need to open-write-close in each loop. Do i still need to flush the data?

---

If you really care about corruption due to a crash, I recommend a network-based solution that is independent of the test system. Windows may decide to simply ignore calls to flush a file. Even worse, your hard disk controller (which has a buffer and controller of its own) may decide to ignore calls for a hard flush as well. You can tell the controller that you're going to turn it off, which should scare it into committing the writes, but I would not call that a solution. On a network, you can be reasonably sure that you successfully handed off the data to a system that is better equipped to deal with the robustness requirement.