10-13-2020 04:11 PM
Hello!
I am trying to understand all the ways to stream data to disk using LabVIEW. Right now I am just using 'Write to 2D text File.vi' and I would like to switch so I can truly stream data to disk while the .vi is running.
My question is specifically regarding the LabVIEW Example .vi called 'Buffered Stream to Tab-Delimited Text File.vi'.
It appears to me that they are opening/creating a file, then using 'Write to Binary File.vi', and then also 'Write to Text File.vi' at the end of the execution.
When the file is completed, it is saved as a tab-delimited file and easily opened. i.e. it is not saved as a binary file.
Why is it possible to use 'Write to Binary File.vi' when creating a 2D text file? Is there some auto-conversion going on behind the scenes?
It also seems like the data is actually only written to disk when the .vi finishes executing. I would think that the streamed buffer means it is written to disk when the buffer is full. But I ran it for 30 seconds and the buffer gets full many times over. I'm not sure what actually happens to the data until it is written to the text file...
I know this is a fairly general question, but none of the help docs I could find about binary or text files seemed to offer light on using them both.
Thanks,
Tom
Solved! Go to Solution.
10-13-2020 04:53 PM
First, a couple of definitions of terminology.
1. Binary. Everything in a PC is binary. Data consist of 1 or more bytes that are 8 bits in length.
2. Text file. A text file is also a kind of binary file, but a special kind where all the characters are human readable as defined by the ASCII table. (Well, some are not such as carriage returns, line feeds, tab characters).
The example you are showing does have a Write to Binary in it. But it takes a bunch of strings, converts them to U8, and then writes the U8's to a file. So instead of writing the letter "A" for instance, the string to U8 is converting it to the unsigned 8-bit integer (U8), 65 decimal. That is the ASCII value for the character A. Then it writes that U8 integer out as "binary" to the file. It is a byte. Guess what happens if you write the "A" using Write to Text File? The OS converts that A to 65 and writes that same byte out to the file.
Either way, the value on the disk is 65, but if you open that file up in notepad, you can read the letter "A"
I'm not sure why they wrote the example that way, but they did put a clue in the comments.
"Note: There are no excess memory allocations in the loop. This is because the In Place Element Structure is used with U8 arrays, as opposed to using the String Subset function." So I think it was for better memory management to handle do more easily manipulate the "string" of bytes by handling them as arrays.
So to finish off, you say "When the file is completed, it is saved as a tab-delimited file and easily opened. i.e. it is not saved as a binary file." But it IS saved as a binary file. All files are binary. It just so happens that all the bytes in that file just happen to be human-readable when you open it up in notepad based on the ASCII table.
10-13-2020 06:29 PM
Thank you for the quick and thorough reply. Definitely answers my original question.
I do understand that all files are binary. I suppose what I should have said was 'a human readable file' versus 'non-human readable binary file'. I am thinking of the binary representation of data which takes less memory than an ASCII file.
I have a follow-up question (and let me know if you think I should put in a new question).
When the example says 'stream to disk', do you know if it is actually streaming to hard disk memory while the .vi is running? Or is it simply storing in RAM until execution ends, then dumps it to the hard disk?
I've read a decent amount about what streaming to disk truly means, and it sounds like the best way to stream to disk instead of saving in RAM is to use 'Write to Binary File.vi' or TDMS files. I thought that the example (since it used the explicit binary file .vi) would fill the file as it ran, but it doesn't seem to be doing so.
10-13-2020 06:47 PM
The data is going directly to the disk cache and then written to the drive. It is not using up a ton of RAM while accumulating data.
10-13-2020 07:17 PM
Thank you, crossrulz.
So to put a bow on it, is the following accurate:
> file streaming with binary or TDMS files writes data to disk cache until closed, then to drive upon closing, and doesn't use a lot of RAM
> writing to file using methods like '2D Text File' writes to RAM, uses a lot of memory, then writes to drive upon closing properly?
Really appreciate ya.
-Tom
10-13-2020 09:25 PM - edited 10-13-2020 09:32 PM
@tsr26 wrote:
Thank you, crossrulz.
So to put a bow on it, is the following accurate:
> file streaming with binary or TDMS files writes data to disk cache until closed, then to drive upon closing, and doesn't use a lot of RAM
> writing to file using methods like '2D Text File' writes to RAM, uses a lot of memory, then writes to drive upon closing properly?
Really appreciate ya.
-Tom
Point #1. Not necessarily true. The disk cache will write out the data whenever it feels it needs to. It doesn't wait until you close the file. Of course closing the file in software will be a signal to the drive to go out and write out whatever data is remaining in the cache.
Point #2. Saying what takes a lot of memory or less has nothing to do with the file type. You can stream ASCII type files to disk. So instead of building large 2-D string arrays, you can build them in smaller chunks, or even row by row. Likewise you could build up a large amount of binary data before you feed it to the File Write.
10-14-2020 01:04 AM
@RavensFan wrote:
Point #1. Not necessarily true. The disk cache will write out the data whenever it feels it needs to. It doesn't wait until you close the file. Of course closing the file in software will be a signal to the drive to go out and write out whatever data is remaining in the cache.
In addition to what RavensFan wrote: There is a 'Flush File' node in the 'Advanced File Function' palette.
From the function help:
Writes all buffers of the file identified by refnum to disk and updates the directory entry of the file associated with refnum.
The file remains open, and refnum remains valid. Data written to a file often resides in a buffer until the buffer fills up or until you close the file. This function forces the operating system to write any buffer data to the file.
10-15-2020 03:04 PM - edited 10-15-2020 03:07 PM
I thought I would toss in a couple of generalities I have come to realize.
Frankly I have never had a reason to use anything other than a text file for saving data.