What could be the best file format to save data for my application?

[charmelony_]

Hello,

In my experiments, I measure different type of outputs with respect to given inputs. In the first figure, you can have some idea about the data output. This is only for one channel of the rig, I have up to 20 channels by using multiple test rigs. In the second figure, block diagram of save VI can be seen. Data are saved to normal file with intervals that I can decide. Experiments will continue couple of months -maybe even a year(s)-. 

I was updating the VI scripts and could not decide whether I should stick the current saving format, or a different one e.g. tdms. All data will be further processed with respective softwares. The VI has an automatic control function as well. I am going to have lots of long data files. I do not want to lose any data point (only related with saving) and I need to access to the data file while the VI is working (currently I use notepad++ for that). Do you suggest me to switch tdms format?

[RTSLVU]

I have had long term tests running for over a year collecting >50 data points at 1 second intervals.

Frankly, I have found no need for anything more complex than a tab delimited text file.

I found it more important to create a new file daily to limit the file size and put the date in the file name so it's easy to go back and find data from a certain date.

Oh and a note about viewing data files while collecting data...

DON'T

1. Open the file
2. Write a line of data
3. Close the file

I know this makes it easy to look at the data file while your program is running but... 

It you have the data file open in Notepad++ or whatever at the same time LabVIEW tries to write to the file.

Your LabVIEW program will crash with a file sharing violation... (The file will be locked as it is open in Notepad++)

DO

1. Open your file at the beginning of your program
2. Pass the File Reference around your loop with a shift register
3. Write data as needed
4. Close the file at the end of your program

Windows will lock the file so not other program can write to it.

If you need to view the file while your program is running.

Either make a copy and open the copy or most programs (like Excel) will notice the file lock and offer to open the file read only or open a copy automatically.

[JÞB]

With the requirement for concurrent access I am going to recommend TDMS.  You are just plain going to have better experience with having the measurment data as actual numeric data types than text anyway and any spreadsheet file will be far superior for analysis than any text editor. 

As a bonus TDMS also can log events.  Since you mentioned that there are automated controls as well logging changes and user comments as events can be a huge value add to the file.

[RTSLVU]

---

@JÞB wrote:

With the requirement for concurrent access I am going to recommend TDMS.  You are just plain going to have better experience with having the measurment data as actual numeric data types than text anyway and any spreadsheet file will be far superior for analysis than any text editor. 

As a bonus TDMS also can log events.  Since you mentioned that there are automated controls as well logging changes and user comments as events can be a huge value add to the file.

---

You can write numeric data to a text file and open in in Excel for analysis just fine.

I have been doing it for >20 years

Where as proprietary file formats run the risk of corruption and loss of all data. 

[JÞB]

---

@RTSLVU wrote:

---

@JÞB wrote:

With the requirement for concurrent access I am going to recommend TDMS.  You are just plain going to have better experience with having the measurment data as actual numeric data types than text anyway and any spreadsheet file will be far superior for analysis than any text editor. 

As a bonus TDMS also can log events.  Since you mentioned that there are automated controls as well logging changes and user comments as events can be a huge value add to the file.

---

You can write numeric data to a text file and open in in Excel for analysis just fine.

I have been doing it for >20 years

Where as proprietary file formats run the risk of corruption and loss of all data. 

---

Excel is a proprietary file format.  TDMS is published.   TDMS is made for test measurement data and allows concurrent access. txt and csv do not  TDMS has an addon for Excel and several other free viewers. xlxs does not. 

[charmelony_]

---

@RTSLVU wrote:

I have had long term tests running for over a year collecting >50 data points at 1 second intervals.

Frankly, I have found no need for anything more complex than a tab delimited text file.

I found it more important to create a new file daily to limit the file size and put the date in the file name so it's easy to go back and find data from a certain date.

Oh and a note about viewing data files while collecting data...

DON'T

1. Open the file
2. Write a line of data
3. Close the file

I know this makes it easy to look at the data file while your program is running but... 

It you have the data file open in Notepad++ or whatever at the same time LabVIEW tries to write to the file.

Your LabVIEW program will crash with a file sharing violation... (The file will be locked as it is open in Notepad++)

DO

1. Open your file at the beginning of your program
2. Pass the File Reference around your loop with a shift register
3. Write data as needed
4. Close the file at the end of your program

Windows will lock the file so not other program can write to it.

If you need to view the file while your program is running.

Either make a copy and open the copy or most programs (like Excel) will notice the file lock and offer to open the file read only or open a copy automatically.

---

Thank you for the suggestion. One interesting note, I have never been encounter any error/crash or lost data by reading the data file with notepad++ while the vi is running. I tried your suggestion, currently, the vi creates a data file and then immediately starts writing (no open-close). Even with this option, I can still use notepad++ without any problem. Do I miss something?

[santo_13]

In my experience, only EXCEL holds the opened file reference and causes an error whereas opening the file using Notepad or Notepad++ in parallel to your logger does not cause any sort of issue.

[crossrulz]

---

@charmelony_ wrote:
Thank you for the suggestion. One interesting note, I have never been encounter any error/crash or lost data by reading the data file with notepad++ while the vi is running. I tried your suggestion, currently, the vi creates a data file and then immediately starts writing (no open-close). Even with this option, I can still use notepad++ without any problem. Do I miss something?

---

Notepad++ and UltraEdit open the files without locking them.  I'm not as familiar with Notepad++, but UltraEdit will check to see if the file has been updated when you reactivate the window and ask if you want to view the updated file.  As long as you don't save through those interfaces, you are fine.

Where I have had issues before was opening test data files in Excel while running a test.  Excel (and I assume Word) lock the files.  This cause errors with my test.  Granted, the data file was being opened, written, and closed for each new data line (not my design choice).

[santo_13]

---

@crossrulz wrote:

---

@charmelony_ wrote:
Thank you for the suggestion. One interesting note, I have never been encounter any error/crash or lost data by reading the data file with notepad++ while the vi is running. I tried your suggestion, currently, the vi creates a data file and then immediately starts writing (no open-close). Even with this option, I can still use notepad++ without any problem. Do I miss something?

---

Notepad++ and UltraEdit open the files without locking them.  I'm not as familiar with Notepad++, but UltraEdit will check to see if the file has been updated when you reactivate the window and ask if you want to view the updated file.  As long as you don't save through those interfaces, you are fine.

---

Based on your description of UltraEdit, Notepad++ also works the same way, whenever you switch the active window, it checks for modification and provides the option to reload the file.

[MichaelS78]

Yes excel , word etc.. lock files,npp does not. It even has a tail function to watch active files and update as they are written.

For very long tests I agree with using text files and writing a new file daily but also maybe consider a database?

How robust are tdms files that don't get closed properly?  That is an interesting point because long running tests are more likely to suffer a crash/power failure.