LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Writing to file using both Write to Binary and Write to Text File

Solved!
Go to solution

Hello!

I am trying to understand all the ways to stream data to disk using LabVIEW. Right now I am just using 'Write to 2D text File.vi' and I would like to switch so I can truly stream data to disk while the .vi is running.

 

My question is specifically regarding the LabVIEW Example .vi called 'Buffered Stream to Tab-Delimited Text File.vi'.

It appears to me that they are opening/creating a file, then using 'Write to Binary File.vi', and then also 'Write to Text File.vi' at the end of the execution.

When the file is completed, it is saved as a tab-delimited file and easily opened. i.e. it is not saved as a binary file.

 

Why is it possible to use 'Write to Binary File.vi' when creating a 2D text file? Is there some auto-conversion going on behind the scenes?

It also seems like the data is actually only written to disk when the .vi finishes executing. I would think that the streamed buffer means it is written to disk when the buffer is full. But I ran it for 30 seconds and the buffer gets full many times over. I'm not sure what actually happens to the data until it is written to the text file...

 

I know this is a fairly general question, but none of the help docs I could find about binary or text files seemed to offer light on using them both.

 

Thanks,

Tom

0 Kudos
Message 1 of 8
(2,836 Views)
Solution
Accepted by tsr26

First, a couple of definitions of terminology.

 

1. Binary.  Everything in a PC is binary.  Data consist of 1 or more bytes that are 8 bits in length.

2. Text file.  A text file is also a kind of binary file, but a special kind where all the characters are human readable as defined by the ASCII table.  (Well, some are not such as carriage returns, line feeds, tab characters).

 

The example you are showing does have a Write to Binary in it.  But it takes a bunch of strings, converts them to U8, and then writes the U8's to a file.  So instead of writing the letter "A" for instance, the string to U8 is converting it to the unsigned 8-bit integer (U8), 65 decimal.  That is the ASCII value for the character A.  Then it writes that U8 integer out as "binary" to the file.  It is a byte.  Guess what happens if you write the "A" using Write to Text File?  The OS converts that A to 65 and writes that same byte out to the file.

 

Either way, the value on the disk is 65, but if you open that file up in notepad, you can read the letter "A"

 

I'm not sure why they wrote the example that way, but they did put a clue in the comments.

"Note: There are no excess memory allocations in the loop. This is because the In Place Element Structure is used with U8 arrays, as opposed to using the String Subset function."  So I think it was for better memory management to handle do more easily manipulate the "string" of bytes by handling them as arrays.

 

So to finish off, you say "When the file is completed, it is saved as a tab-delimited file and easily opened. i.e. it is not saved as a binary file."   But it IS saved as a binary file.  All files are binary.  It just so happens that all the bytes in that file just happen to be human-readable when you open it up in notepad based on the ASCII table.

Message 2 of 8
(2,808 Views)

Thank you for the quick and thorough reply. Definitely answers my original question.

 

I do understand that all files are binary. I suppose what I should have said was 'a human readable file' versus 'non-human readable binary file'. I am thinking of the binary representation of data which takes less memory than an ASCII file.

 

I have a follow-up question (and let me know if you think I should put in a new question). 

When the example says 'stream to disk', do you know if it is actually streaming to hard disk memory while the .vi is running? Or is it simply storing in RAM until execution ends, then dumps it to the hard disk?

I've read a decent amount about what streaming to disk truly means, and it sounds like the best way to stream to disk instead of saving in RAM is to use 'Write to Binary File.vi' or TDMS files. I thought that the example (since it used the explicit binary file .vi) would fill the file as it ran, but it doesn't seem to be doing so.

0 Kudos
Message 3 of 8
(2,788 Views)

The data is going directly to the disk cache and then written to the drive.  It is not using up a ton of RAM while accumulating data.


GCentral
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
"Not that we are sufficient in ourselves to claim anything as coming from us, but our sufficiency is from God" - 2 Corinthians 3:5
Message 4 of 8
(2,778 Views)

Thank you, crossrulz.

 

So to put a bow on it, is the following accurate:

 

> file streaming with binary or TDMS files writes data to disk cache until closed, then to drive upon closing, and doesn't use a lot of RAM

> writing to file using methods like '2D Text File' writes to RAM, uses a lot of memory, then writes to drive upon closing properly?

 

Really appreciate ya.

-Tom

0 Kudos
Message 5 of 8
(2,763 Views)

@tsr26 wrote:

Thank you, crossrulz.

 

So to put a bow on it, is the following accurate:

 

> file streaming with binary or TDMS files writes data to disk cache until closed, then to drive upon closing, and doesn't use a lot of RAM

> writing to file using methods like '2D Text File' writes to RAM, uses a lot of memory, then writes to drive upon closing properly?

 

Really appreciate ya.

-Tom


Point #1.  Not necessarily true.  The disk cache will write out the data whenever it feels it needs to.  It doesn't wait until you close the file.  Of course closing the file in software will be a signal to the drive to go out and write out whatever data is remaining in the cache.

 

Point #2.  Saying what takes a lot of memory or less has nothing to do with the file type.  You can stream ASCII type files to disk.  So instead of building large 2-D string arrays, you can build them in smaller chunks, or even row by row.  Likewise you could build up a large amount of binary data before you feed it to the File Write.

Message 6 of 8
(2,743 Views)

@RavensFan wrote:

Point #1.  Not necessarily true.  The disk cache will write out the data whenever it feels it needs to.  It doesn't wait until you close the file.  Of course closing the file in software will be a signal to the drive to go out and write out whatever data is remaining in the cache.

In addition to what RavensFan wrote: There is a 'Flush File' node in the 'Advanced File Function' palette.

 

From the function help:

Writes all buffers of the file identified by refnum to disk and updates the directory entry of the file associated with refnum.
The file remains open, and refnum remains valid. Data written to a file often resides in a buffer until the buffer fills up or until you close the file. This function forces the operating system to write any buffer data to the file.

 

0 Kudos
Message 7 of 8
(2,712 Views)

I thought I would toss in a couple of generalities I have come to realize. 

 

  1. Using the primitives that pass a "file reference" rather than a "file path" are faster because they keep the file open between writes
    1. The "Write Delimited Spreadsheet" and Express VI's Open the file, write to it, and close the file on every single write.
  2. This method has the benefit of locking the file so no other program can open it. 
    1. The "Write Delimited Spreadsheet" and Express VI's do not lock the file between writes and if another program has the file open (like someone looking at the data) your LabVIEW program will crash due to a Windows File Sharing violation when it tries to write to it.

 

Frankly I have never had a reason to use anything other than a text file for saving data.

========================
=== Engineer Ambiguously ===
========================
Message 8 of 8
(2,635 Views)