LabVIEW runs out of memory when I'm writing to a spreadsheet file

[dNr]

Hi,

Somebody had helped me previously here relating to opening up a binary file and writing the data to a text file. I've modified it slightly, and for smaller files (such as the one included in the zip file), it works fine. However, I have a LOT 120MB binary files (yeah, they are big) which I need to convert into text files, and LabVIEW runs out of memory. I've looked about in the help files, and I've tried a few of the suggestions (like a subvi) from the help files, but every time I end up with LabVIEW running out of money (see bmp files in the zip file).

I've tried only running LabVIEW to try and allow it to have as much memory as possible, but that doesn't seem to help unfortunately. 

If anyone has any suggestions, I'd appreciate hearing them. 

dNr

P.S. Zip file = VI for changing binary to ASCII

Then: example of a small binary file (that will in fact run without problem), screen shots of the error messages.

[JÞB]

Alright,

Too much data at one time.  You will need to manage the amount of data and the timing.  (where's Christian when you need him?)

The files are huge! and you cannot hold them all in memory especially when you copy the data like LabVIEW likes to do when you create a node (tunnel, control, indicator, wire branch etc...)

The easiest way to stop this memory overload is to read and write in sections.  (read a Mbyte-write it and do it again) that will only take a few meg of memory to do (and a lot of time allocating and de-allocating memory). 

I haven't had time or access to your sub-vis to fully annalize your memory multipliers but there is a good knowledge base on handling large data sets.  Google it please (Iwill learn how to post links eventually)

[smercurio_fc]

Jeff is absolutely correct. You cannot read all of that data all at once. The VI you were given by Ravens Fan was not intended to be used for such large datasets. You will need to use a loop to read the data in chunks. Since you are reading records at a time you can specify a certain number of records to read (change the -1 input to the Read from Binary File), convert them, and write them to file. Repeat and rinse. The Write to Spreadsheet File has an "append to file?" input so you can put the read/convert/write in the same loop.

[dNr]

Smercurio_fc & Jeff,

Thanks for the sugguestions. I will try set up a loop so I can read in a bit at a time and then convert them.

I realize trying to read in such large files is sort of insane, but I cannot change how the data I'm trying to analyze is being collected. I've voiced my concern about this, so I'm hoping in the future people will change their collection mechanism! 

Thanks,

dNr

[dNr]

So I thought a While loop would probably be best suited for the purposes here, but now I'm not so sure (see png file). The thing that I'm confused about is that I set the "bytes to read" to 1000000, which is definitely smaller than the binary file size (~120MB), however, I end up getting an error message saying that it's hitting the end of the file. I suppose that's good because it's reading to the end of the file to convert to ASCII, and if I tell it to continue, it chugs away until it runs out of memory again.

If I was doing this in Matlab, I have a good idea what I'd do, but since we're not working in Matlab for the main data collection (my collegue that is generating these binary files doesn't really even know the binary format so I could attempt to read it in via Matlab),  I'm paddling up the raging river that is LabVIEW without a proper paddle or boat for that matter!

Thanks for the suggestions so far. My apologizes that I'm not nearly as familiar with this language as I really should be.

dNr

[smercurio_fc]

You are not using the Read From Binary File function properly. Please read the LabVIEW Help for that function. The "count" input does not refer to the number of bytes, unless you explicitly wire a U8 to the "data type" input. From the Help:

data type sets the type of data the function uses to read from the binary file. The function interprets the data starting at the current file position to be count instances of data type. If the type is an array, string, or cluster containing an array or string, the function assumes that each instance of that data type contains size information. If an instance does not include size information, the function misinterprets the data. If LabVIEW determines that the data does not match the type, it sets data to the default for the specified type and returns an error. 

count is the number of data elements to read. Data elements can be bytes or instances of data type. The function returns count data elements in data, or if it reaches the end of the file, it returns all the complete data elements read thus far and an end-of-file error. By default, the function returns a single data element. If count is –1, the function reads the entire file. If count is less than –1, the function returns an error.

If count calls for an array of elements and the specified data type is an array, the function automatically returns a cluster of arrays or cluster array because LabVIEW does not allow arrays of arrays.

The important part is that you are wiring a datatype. This means the "count" refers to the number of records, not the number of bytes. This means that you have to specify a certain number of records to read, not a certain number of bytes. Also, it makes no sense to stop the while loop based on a Stop button on the front panel. What's the point of continuing to read the file if there's nothing left? You should open the file outside the loop (opening each time is pointless), read x number of records, convert, write, and repeat until there's nothing left. You can stop the loop on an error from the Read From Binary File function, which will occur when it gets an EOF. Then, close the file (outside the loop). 

P.S. Do you have a (small) sample of a file that you read? 

Message Edited by smercurio_fc on 08-03-2009 05:29 PM

[dNr]

Oh! I must have read that incorrectly then-- I've been looking at LabVIEW all day and I can't make heads from tails this moment....

I guess then I should change the count number to 3600 as there are 3600 records per large bin file... but now I'm not sure. I'm feeling more confused about this by the moment. The data type is already constructed for the time-stamp information.

My impression from Jeff's comment was that the open/read could be included in the loop. I thought that was odd, but that's what I did.

There is an example (small) of the bin files I'm working with is in one of the two zip files I included at the start of this thread. I thought it would be stupid to include the vi without an example input file.  The original binary to ascii works well with that example bin, it's just the large files it's choking on.

[JÞB]

S

---

dNr wrote:

Oh! I must have read that incorrectly then-- I've been looking at LabVIEW all day and I can't make heads from tails this moment....

I guess then I should change the count number to 3600 as there are 3600 records per large bin file... but now I'm not sure. I'm feeling more confused about this by the moment. The data type is already constructed for the time-stamp information.

My impression from Jeff's comment was that the open/read could be included in the loop. I thought that was odd, but that's what I did.

There is an example (small) of the bin files I'm working with is in one of the two zip files I included at the start of this thread. I thought it would be stupid to include the vi without an example input file.  The original binary to ascii works well with that example bin, it's just the large files it's choking on.

---

My appologies for mis-stating the suggestion.

DO open referances to files, and close them, OUTSIDE the read and process structure (or a process and write structure).  Just good programming style.

[smercurio_fc · Accepted solution]

---

dNr wrote: 

I guess then I should change the count number to 3600 as there are 3600 records per large bin file... but now I'm not sure. I'm feeling more confused about this by the moment. The data type is already constructed for the time-stamp information.

---

No, you're still not understanding. If the file contains 3600 records, then telling it to read 3600 records at once simply does exactly the same thing as what you had before with the -1 wired to the count. The -1 tells the function to read all records. That's precisely what you don't want to do. You want to read a limited number of records at a time. You do this in a loop until you've read all the records.

You also do not want to open the file each time in the loop. You want the read inside the loop since that's what you're doing repeatedly.

This has nothing to do with LabVIEW. It's basic programming principles. 

Please see attached example of how you can do this. Note that I used code to create the filename based on the first timestamp. Since you are reading the data in chunks you don't want to change the filename along the way because the timestamp can change. 

[dNr]

smercurio_fc,

Ah. Ok. Now that I'm looking at the vi you attached, what you were saying makes more sense. As I said before, I have a good idea how I'd do this operation in Matlab, but it seems that LabVIEW really throws me for a loop (pun intended). Honestly, the visualization of for and while loops is not exactly my slice of cake... *sigh* Then again, I'm a chemist, not a computer programer, so forgive me for my lack of understanding and miscommunication!  Either way, I am indebted to your vi you constructed, it does what I need it to do and without causing LabVIEW to choke on data!

Jeff,

No worries. I may not have been very clear in my initial post you replied to.

Many Thanks,

dNr

Message Edited by dNr on 08-04-2009 08:35 AM