LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Memory problem when reading in a binary file

Highlighted

I am trying to read in a binary file that is about 300 MB.  At the read file vi, I get an error message popping up saying that I do not have enough memory to complete the operation.  The system has 2 GB memory.  Files that are 150MB read in perfectly fine, with no system lag.  Monitoring physical memory usage, hardly any is being used.  Does LabVIEW not recognize the correct amount of memory in my system, or is there some other problem causing me to not be able to read in a large file?

 

Thanks,

JR

0 Kudos
Message 1 of 14
(1,206 Views)

Hi JR,

 

LabView needs continuous chunks of memory for arrays. This may be the cause for such errors...

 

You may:

- work on parts of the file

or:

- init an array the size of the file, then load smaller parts of the file in a loop and replace array subsets

Best regards,
GerdW
CLAD expired, using LV2011SP1 + LV2017 (+LV2019 sometimes) on Win7+cRIO
Kudos are welcome Smiley Wink

0 Kudos
Message 2 of 14
(1,200 Views)

Im still sort of surprised that the memory error happens between 150 and 300 Mb. LabVIEW is supposed to be able to read in 2 gig through File I/O with no problems.

 

2 gig KnowledgeBase

Chris Van Horn
Applications Engineer
0 Kudos
Message 3 of 14
(1,198 Views)

Hi JRaetz,

 

What kind of binary file and how are you reading it?

 

Watch out for data copies using the Tools>Profile>Buffer Allocation tools. Three hundred MB gets of data becomes very large if it is copied a couple of times.

 

This link might help also:  http://zone.ni.com/devzone/cda/tut/p/id/3625 

 

Please post a block diagram image or better still the VIs that you think are causing the problem. 

 

 Regards

 Ruben 

 

 

 

0 Kudos
Message 4 of 14
(1,192 Views)

It is a 10 by 8,000,000 2D array of singles, written to a binary file using Write to Binary File, and read back using the related Read from Binary File from the File I/O pallet.  The file size comes out to be10*201*200*200*4bytes/sgl = 314,062 KB.

 

If it was a problem with not having that size of continuous memory, wouldn't it attempt to reorganize memory or use virtual memory?

0 Kudos
Message 5 of 14
(1,168 Views)

Hi JR,

 

I said "maybe" - but that's the most common source of problems when handling (really) big arrays.

 

Still my advice would be to load smaller parts of the file and use them to replace data in a pre-allocated array - if you really need that much data at once in memory.

Best regards,
GerdW
CLAD expired, using LV2011SP1 + LV2017 (+LV2019 sometimes) on Win7+cRIO
Kudos are welcome Smiley Wink

0 Kudos
Message 6 of 14
(1,162 Views)

Gerd-

 

Can you tell me how I can read part of a binary file instead of the entire thing?  It's definitely an option worth trying, but I can't figure out how to do it.

 

Thanks,

JR

0 Kudos
Message 7 of 14
(1,153 Views)

Hi JR,

 

did you notice the context help window? At least for LV7.1 it will provide the neccessary information:

Smiley Wink

Do you notice the inputs for number of rows/columns? And the read offset, which you have to wire when reading chunks from a file?

Message Edited by GerdW on 03-31-2009 05:37 PM
Best regards,
GerdW
CLAD expired, using LV2011SP1 + LV2017 (+LV2019 sometimes) on Win7+cRIO
Kudos are welcome Smiley Wink

0 Kudos
Message 8 of 14
(1,136 Views)
I also recommend taking a dive into the code for "Read from SGL file" Then doing this kind of file operations you do your processing in a loop. Before you enter the loop you open the file, and you close it after the loop. I do not have time for an example now, but I can give you one if you want


Besides which, my opinion is that Express VIs Carthage must be destroyed deleted
(Sorry no Labview "brag list" so far)
0 Kudos
Message 9 of 14
(1,118 Views)

Thanks to everyone responding and trying to help.

 

I am using LabVIEW 8.6, and I cannot find a Read from SGL File block.  I have found through subsequent testing that  I can read the file in in 3 separate parts, but only once.  After running the file to read it into 3 sub-arrays, if I try to run it again it will produce a memory error again.  Shutting down and restarting LabVIEW fixes it again.  Monitoring my memory usage, when I can succesfully read into 3 arrays, I jump from ~400MB of used memory to just under 1 GB.  The second time, when reading the file does not work, I'm up to 1.2 GB used, of my 2GB available.  I don't know if any of that is helpful or not, but I thought I'd add it.

 

What do you mean by taking a dive into the code for the block?  I'm a C programmer who started using LabVIEW 3 months ago, so if I could see what was going on behind the scenes it would help, but I don't know how to do that.

 

Thanks again,

JR

0 Kudos
Message 10 of 14
(1,104 Views)