Reading large binary files into an array for parsing

[id]

I have a large binary log file, consisting of binary data separted by header flags scattered nonuniformly thorughout the data.  The file size is about 50M Byte.  When I read the file into an array, I get the Labview Memory full error.  The design of this is to read the file in and then parse it fro the flags to determine where to separate the data blocks in the byte stream. 

There are a few examples that I have read on this site but none seem to give a straight answer for such a simple matter.   Does anyone have an example of how I should approach this?

[RavensFan]

50 MB is not that big.  You'd really have to show us a small section of the file so we can see how it is set up (such as what the header flags look like), along with your VI that is reading it.

[id]

Lets start with just this fundamental allocation of an arry that is 50M.  This is what I get.

[GerdW]

Hi id,

why would you need an array of 50M doubles - which takes 400MB of RAM?

Wouldn't an array of U8 be sufficient for your file?

And why do you need to initialize an array when you want to read from file?

[Ray.R]

I agree with Gerd.  If you are working with binaries, why not use U8 instead of doubles.

If the file is indeed 50MB, then the array should be expecting 52428800 elements, not 50000000.  So if you read the file in a loop and populate an element at a time, you could run out of memory fast because any additional element insertion above 50000000 may require additional memory allocation of the size above 50000000 (potentially for each iteration).  This is just speculation since I don't see the portion of your code that populates the array.

Question:  Why do you need an array?  What do you do with the data after you read it?  I agree with Altenbach, 50MB is not that big, so working with a file of such a size should not be a problem.

[johnsold]

You indicate that you want to separate blocks of data separated by the headers.  What is the size of the largest block which will be contained in the file?  Perhaps you could read segments of that size and parse out the headers.  Then retain the second header and any following bytes in a shift register to add to the next block.

As with the other responders, I agree that you should be able to read the whole file.  The technique I outlined above should work on nay size file, provided the block of data will fit in memory.

Lynn

[id]

The array is decoded with a complex data decoder ring after it is read into memory.  It is user interface selectable as to how to decode it.

[GerdW]

Hi id,

as you didn't ask any further questions we might think your problem is solved...

Otherwise attach your VI so we can have a peek into it!

[Ray.R]

How is the data passed to the decoder(s)?

How does the decoder process the data?

With more details, we can provide more appropriate suggestions.

or

Post the code as Gerd suggested.

[Ben]

Don't read the whole file at once.

Open the file as a binary file and just start reading a byte at a time until you get to what you are looking for. Then read the number of bytes you need starting at the file pointer indicated by the data foramtting of the file and then pass that sub-set of data to your "secret Decoder Ring" to process.

Ben