Read large binary files over 1000 records

[Richard_Z.]

I have some large binary data files created in Labview that cannot be fully read in Labview. Normally we acquire data in Labview and analyze in Matlab. However now we have the need to read them in Labview as well. The files have a header, which we can read just fine. Then there is data from 64 channels. There is a data element from each channel, for each time stamp, for 300,000 bins. So the array of data read in should be 64 rows by 300,000 columns.

Matlab reads these files in their entirety. But Labview will only read the first 1000 columns. After I read the data file my array is only 64 rows by 1000 columns. It seems like either the default maximum array dimension or the maximum number of records for Read from Binary File.vi is some how limited. I searched around and could not find an answer.

Attached is my vi. The data file is too large to post to the forum. If some one can tell me how to post large files I will put it up.

Any help would be greatly appreciated!

[RavensFan]

Can you post a smaller version of your binary file?  What is the layout of the header for these files?

From what I can see from your code is that you descriptions for what you are doing don't match what you are actually doing.

You say, "Determine header size by reading to the second header entry which is the header size.".  But in reality, you are only reading 1 count of the U8 datatype.  That doesn't sound like you are "reading to the second header entry."

Then you say "Re-read the file to end of header using the size from the previous step.".   But you aren't rereading anything.  You aren't using any information from the previous step.  You just read 1 count of an I32.  So that is 4 bytes.

You have now read 5 bytes total, and those 4 remaining determine how many more pieces of information you are going to read.  So you now read the amount from step 2, add 5 to that and then read that many elements of a 2D I16 array.  (I'm not even sure that is the correct way to read a 2-D array.)

You also have several "Deny" functions in there, do you need them?  YOu have several Get File Size functions, but you never use any data from them.  You are also using hidden controls to define datatypes.  Just convert them to constants to define the datatypes so you don't have unnecessary controls on the front panel.  You should also close the file reference when you are done.

In the end, the steps you are doing don't seem to match what you are describing.  My gut feel is that you don't have a handle on what your header part actually looks like, and you are reading the wrong parts of it to determine how much data to read.  Understand how your header is written.  Extract the correct amount of data to read, and make sure you read through the header so that your file pointer is at the point where the data actually starts before reading the data.  Put indicators on all of the parts where you are reading header information to be sure you are actually reading what you think you should be reading.

[GerdW]

Hi Richard,

The data file is too large to post to the forum. If some one can tell me how to post large files I will put it up.

We don't want big data files here…

- Why do you "Deny access" in this VI?

- Why do you close/reopen the file in between?

- Why don't you state the number of element to read with the last Read function?

What's the format of your file (and especially the header)?

Let's start easy:

(Instead of reading one U8 and one I32 you could also read five U8 values…)

[Richard_Z.]

Thanks for the quick response. I have uploaded a sample date file here http://wikisend.com/download/599870/zip.zip

The “Deny Access” comes directly from the “Read Binary File.vi” example file. I’ll try deleting it but I doubt that is the problem.

The header size is not the same for all files. I open the file, read the header to read the entry for the size of the header. Then close the file and re-open it and skip forward to the end of the header and start reading the data.

I could not find a clean way to read the header entry from the header size and then skip to the end of the header without closing/re-opening it.

I can plot the data from the first 1000 column and it looks fine. It just stops reading after that.

[RavensFan]

Get rid of the Deny Access.  I doubt it is the problem also, but there is no need for it.

"The header size is not the same for all files. I open the file, read the header to read the entry for the size of the header".

When you read the header size, then you should use that information.  Some way or another you need to figure out how many bytes are in the header, and where the those bytes are.  Right now you are assuming, you have a single byte that you aren't paying attention to.  Then 4 more bytes that are an I32 value.  Then everything after those 5 bytes, then you read those 5 bytes plus some as a string, then proceed to read.

Try Gerd's VI snippet.

There is no need to close and reopen the file.  Either read the additional bytes, or use Set File Position to set your file pointer to a specific part of the file.

[crossrulz]

---

@Richard_Z. wrote:

I could not find a clean way to read the header entry from the header size and then skip to the end of the header without closing/re-opening it.

---

Set File Position

[Richard_Z.]

I tried reproducing what you have here (I'm using Labview 8.6) and get:

Error 116 occurred at Read from Binary File in Read Large Binary File Example3.vi

Possible reason(s):

LabVIEW:  Unflatten or byte stream read operation failed due to corrupt, unexpected, or truncated data.

[RavensFan · Accepted solution]

I can't download your data file.  It is blocked by my anti-virus because it is suspicious of your file hosting website.

Do you have your program that creates the datafile?  Seeing how that is written can tell more how the datafile is crated.

Try setting the datatype for the last read to be an I16 constant rather than a 2-D I16 array.  When you read a 2-D array, it expects the row and column size data to be a part of the information it is reading in.  If that information is not there, then it will wind up with an error message like you are getting.

If the data was just written as a series of I16 values, then the row and column sizes won't be there.  So read it in as an I16 datatype and you will get a 1-D array as long as necessary.  Then you would need to reshape that into a 2-D array of the appropriate dimensions.

[Alexander_Sobolev]

Did not find major errors in your VI, can you check and show file save vi?

How do you write header - as string or array? Is 32 bit size written automatically or you first manually write size then string data (that includes size one more time). 

A few corrections, including previous comments

PS False case of the BD below is writing data

PPS: edit I forgot close file. Shame on me.

[Richard_Z.]

I still get the error when it set as a constant. Yes I have the VI that creates the Header, but its very complex and I did not create it. Here is the part that makes the header: