LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Long data logging - How to avoid labview crashs

Solved!
Go to solution
Highlighted

HI!

 

I'm trying to estimate the size of a TDMS file to avoid my application to crash.

I will have to log data for over 10 days straight, and even if there won't be a lot of measurements (2 to 12 per minutes) I just want to make sure everything will be fine but also to understand a point:

 

When labview creates/open/work on a TDMS file:

  • Does it open the whole file or only part of it?
  • When logging data, should I then use TDMS Flush.vi to force data to be saved in the file in case of a crash?
  • What would be the limiting factor for a big file size? My RAM?

 

I've found this article that is showing how to estimate TDMS file size, so thanks to that, I'd say my biggest file size on a 64bits OS would be something like

File_size= 8[bytes] x 13[channels] x (12[measurements/min] x 60[min] x 24[hours] x 10[days]

FS~= 17MB+metadata only?

Seems rather low (and so the problem wouldn't appear really, still curious about the points above though)

 

Thanks in advance

Vinny

0 Kudos
Message 1 of 8
(269 Views)
Highlighted
Solution
Accepted by topic author VinnyLaTaupe

@VinnyLaTaupe wrote:

When labview creates/open/work on a TDMS file:

  • Does it open the whole file or only part of it?
  • When logging data, should I then use TDMS Flush.vi to force data to be saved in the file in case of a crash?
  • What would be the limiting factor for a big file size? My RAM?

 

I've found this article that is showing how to estimate TDMS file size, so thanks to that, I'd say my biggest file size on a 64bits OS would be something like

File_size= 8[bytes] x 13[channels] x (12[measurements/min] x 60[min] x 24[hours] x 10[days]

FS~= 17MB+metadata only?

Seems rather low (and so the problem wouldn't appear really, still curious about the points above tho


1. It opens a file reference with a position pointer. Unless you explicitly read the file it should only be 2 pointers (16 bytes?)

2. I don't think that's necessary, but someone might prove me wrong.

3. File system/OS limitations. Unless you make some glaring mistake the only memory needed should be the buffer of samples before you write the data, and at your mentioned rate it'd be less than 1 sample and write/sec (if data speed is high you often buffer to e.g. 1 sec)

Your size estimation seems correct.

G# - Award winning reference based OOP for LV, for free! ADDQ VIPM Now on GitHub
"Only dead fish swim downstream" - "My life for Kudos!" - "Dumb people repeat old mistakes - smart ones create new ones."
Certified-LabVIEW-Developer
Message 2 of 8
(257 Views)
Highlighted
Solution
Accepted by topic author VinnyLaTaupe

A TDMS file just opens access to the disk and data is streamed to the disk.  The only real size limits are in your harddrive.  Of course, I am assuming you are actually just streaming the data and not building up huge arrays before writing.  You might want to supply some code for us to examine to try to figure out why your code is crashing.  It should not be due to writing to a TDMS file.


GCentral
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
"Not that we are sufficient in ourselves to claim anything as coming from us, but our sufficiency is from God" - 2 Corinthians 3:5
Message 3 of 8
(253 Views)
Highlighted

@crossrulz wrote:

A TDMS file just opens access to the disk and data is streamed to the disk.  The only real size limits are in your harddrive.  Of course, I am assuming you are actually just streaming the data and not building up huge arrays before writing.  You might want to supply some code for us to examine to try to figure out why your code is crashing.  It should not be due to writing to a TDMS file.


Ok good to know, HD limit is pretty high anyway 🙂

I don't have any code nor crash yet, I was anticipating the potential issue, but also was genuinely wondering the points mentionned above.

0 Kudos
Message 4 of 8
(225 Views)
Highlighted

@Yamaeda wrote:

 

1. It opens a file reference with a position pointer. Unless you explicitly read the file it should only be 2 pointers (16 bytes?)

Why 2? 🤔

 

 

0 Kudos
Message 5 of 8
(224 Views)
Highlighted

@VinnyLaTaupe wrote:

@Yamaeda wrote:

 

1. It opens a file reference with a position pointer. Unless you explicitly read the file it should only be 2 pointers (16 bytes?)

Why 2? 🤔

 

 


1 File reference

2 Position within file

G# - Award winning reference based OOP for LV, for free! ADDQ VIPM Now on GitHub
"Only dead fish swim downstream" - "My life for Kudos!" - "Dumb people repeat old mistakes - smart ones create new ones."
Certified-LabVIEW-Developer
0 Kudos
Message 6 of 8
(208 Views)
Highlighted

@VinnyLaTaupe wrote:

HI!

 

I'm trying to estimate the size of a TDMS file to avoid my application to crash.

I will have to log data for over 10 days straight, and even if there won't be a lot of measurements (2 to 12 per minutes) I just want to make sure everything will be fine but also to understand a point:

 

When labview creates/open/work on a TDMS file:

  • Does it open the whole file or only part of it?
  • When logging data, should I then use TDMS Flush.vi to force data to be saved in the file in case of a crash?
  • What would be the limiting factor for a big file size? My RAM?

 

I've found this article that is showing how to estimate TDMS file size, so thanks to that, I'd say my biggest file size on a 64bits OS would be something like

File_size= 8[bytes] x 13[channels] x (12[measurements/min] x 60[min] x 24[hours] x 10[days]

FS~= 17MB+metadata only?

Seems rather low (and so the problem wouldn't appear really, still curious about the points above though)

 

Thanks in advance

Vinny


Are you using DAQmx? If so use the built in logging functions. This is the most efficient way to save data. I have saved data using it for multiple days to a week without any problems. (8 channels 2MSa/s each channel over USB)

 

If you need better efficiency, use the Log Only Mode; however in this mode you won't be able to monitor your signal, it will stream directly to disk. I highly suggest, (it is requirement for Log Only Mode), to have your file size a multiple of the disk sector size. Just find out the sector size and have the number of points in your file be a multiple of it. This works well for disk streaming.

 

Another option, if and only if your program is run as an Administrator, you can have LabVIEW pre-allocate the file. I don't know why you need to be an Admin, but you do. See messy diagram below.

 

Snap47.png

 

 

mcduff

Message 7 of 8
(204 Views)
Highlighted

@mcduff wrote:

Are you using DAQmx? If so use the built in logging functions. This is the most efficient way to save data. I have saved data using it for multiple days to a week without any problems. (8 channels 2MSa/s each channel over USB)

 

mcduff


Which I was, but sadly no. 

Thank you for the tip though.

0 Kudos
Message 8 of 8
(104 Views)