Long data logging - How to avoid labview crashs

[VinnyAstro]

HI!

I'm trying to estimate the size of a TDMS file to avoid my application to crash.

I will have to log data for over 10 days straight, and even if there won't be a lot of measurements (2 to 12 per minutes) I just want to make sure everything will be fine but also to understand a point:

When labview creates/open/work on a TDMS file:

• Does it open the whole file or only part of it?
• When logging data, should I then use TDMS Flush.vi to force data to be saved in the file in case of a crash?
• What would be the limiting factor for a big file size? My RAM?

I've found this article that is showing how to estimate TDMS file size, so thanks to that, I'd say my biggest file size on a 64bits OS would be something like

File_size= 8[bytes] x 13[channels] x (12[measurements/min] x 60[min] x 24[hours] x 10[days]

FS~= 17MB+metadata only?

Seems rather low (and so the problem wouldn't appear really, still curious about the points above though)

Thanks in advance

Vinny

[Yamaeda · Accepted solution]

---

@VinnyAstro wrote:

When labview creates/open/work on a TDMS file:

• Does it open the whole file or only part of it?
• When logging data, should I then use TDMS Flush.vi to force data to be saved in the file in case of a crash?
• What would be the limiting factor for a big file size? My RAM?

I've found this article that is showing how to estimate TDMS file size, so thanks to that, I'd say my biggest file size on a 64bits OS would be something like

File_size= 8[bytes] x 13[channels] x (12[measurements/min] x 60[min] x 24[hours] x 10[days]

FS~= 17MB+metadata only?

Seems rather low (and so the problem wouldn't appear really, still curious about the points above tho

---

1. It opens a file reference with a position pointer. Unless you explicitly read the file it should only be 2 pointers (16 bytes?)

2. I don't think that's necessary, but someone might prove me wrong.

3. File system/OS limitations. Unless you make some glaring mistake the only memory needed should be the buffer of samples before you write the data, and at your mentioned rate it'd be less than 1 sample and write/sec (if data speed is high you often buffer to e.g. 1 sec)

Your size estimation seems correct.

[crossrulz · Accepted solution]

A TDMS file just opens access to the disk and data is streamed to the disk.  The only real size limits are in your harddrive.  Of course, I am assuming you are actually just streaming the data and not building up huge arrays before writing.  You might want to supply some code for us to examine to try to figure out why your code is crashing.  It should not be due to writing to a TDMS file.

[VinnyAstro]

---

@crossrulz wrote:

A TDMS file just opens access to the disk and data is streamed to the disk.  The only real size limits are in your harddrive.  Of course, I am assuming you are actually just streaming the data and not building up huge arrays before writing.  You might want to supply some code for us to examine to try to figure out why your code is crashing.  It should not be due to writing to a TDMS file.

---

Ok good to know, HD limit is pretty high anyway 🙂

I don't have any code nor crash yet, I was anticipating the potential issue, but also was genuinely wondering the points mentionned above.

[VinnyAstro]

---

@Yamaeda wrote: 1. It opens a file reference with a position pointer. Unless you explicitly read the file it should only be 2 pointers (16 bytes?)

Why 2? 🤔

[Yamaeda]

---

@VinnyAstro wrote:

---

@Yamaeda wrote: 1. It opens a file reference with a position pointer. Unless you explicitly read the file it should only be 2 pointers (16 bytes?)

Why 2? 🤔

---

1 File reference

2 Position within file

[mcduff]

---

@VinnyAstro wrote:

HI!

I'm trying to estimate the size of a TDMS file to avoid my application to crash.

I will have to log data for over 10 days straight, and even if there won't be a lot of measurements (2 to 12 per minutes) I just want to make sure everything will be fine but also to understand a point:

When labview creates/open/work on a TDMS file:

• Does it open the whole file or only part of it?
• When logging data, should I then use TDMS Flush.vi to force data to be saved in the file in case of a crash?
• What would be the limiting factor for a big file size? My RAM?

I've found this article that is showing how to estimate TDMS file size, so thanks to that, I'd say my biggest file size on a 64bits OS would be something like

File_size= 8[bytes] x 13[channels] x (12[measurements/min] x 60[min] x 24[hours] x 10[days]

FS~= 17MB+metadata only?

Seems rather low (and so the problem wouldn't appear really, still curious about the points above though)

Thanks in advance

Vinny

---

Are you using DAQmx? If so use the built in logging functions. This is the most efficient way to save data. I have saved data using it for multiple days to a week without any problems. (8 channels 2MSa/s each channel over USB)

If you need better efficiency, use the Log Only Mode; however in this mode you won't be able to monitor your signal, it will stream directly to disk. I highly suggest, (it is requirement for Log Only Mode), to have your file size a multiple of the disk sector size. Just find out the sector size and have the number of points in your file be a multiple of it. This works well for disk streaming.

Another option, if and only if your program is run as an Administrator, you can have LabVIEW pre-allocate the file. I don't know why you need to be an Admin, but you do. See messy diagram below.

mcduff

[VinnyAstro]

---

@mcduff wrote:

Are you using DAQmx? If so use the built in logging functions. This is the most efficient way to save data. I have saved data using it for multiple days to a week without any problems. (8 channels 2MSa/s each channel over USB)

mcduff

---

Which I was, but sadly no. 

Thank you for the tip though.