Somewhat large binary file has trouble being opened

[Anonymous]

Hi,

I'm a new Labview user (Labview 8.0), and I'm trying to convert a
binary file into a wave file and do a lot of other things with that
binary file (like analyzing frequency via a spectrum graph). My file
works fine for files under 150MB, but once the file size is larger than
that, the computer slows way way down and I cannot collect the entire
data set. I already have 1GB of RAM (512x2) and I don't want to pay
more money if the issue can be fixed within Labview.

Is there a way to split up the binary file into little pieces when
being read, or is there another way to go about doing this? I'm using
the File Open dialog to read the file.

If anyone could provide a step-by-step solution to this, I'd be very
grateful. Thanks everyone.

Jennings

[falkpl]

I find it best to work with large files in chunks if possible.  What are you doing with the data?  If it is a serial transformation it is bets to open and read sections of the file for processing, much like a web page loads piecewise to lower latancies of a load.  An additional performance gain is realized when processing smaller arrays which is less demanding on the memory system (fewer page faults, virtual memory useage...)

Paul

[Anonymous]

Here's an update:

I can open the file using the Read from Binary File function, but my
computer says that it runs out of memory after I try to play parts of
the file (I convert the binary file to a wave file and then play it).

Any suggestions would be greatly appreciated.

Jennings

[daveTW]

Hi,
the topic of memory- usage is handled here
http://zone.ni.com/devzone/conceptd.nsf/webmain/732CEC772AA4FBE586256A37005541D3#3
If you load a file of 150MB and do some calculations on it, it is very likely, that there are a lot of data-copys in the memory. You can have a an overview on buffer allocations with Tools»Profile»Show Buffer Allocations (LV8).
Greets, dave

[Anonymous]

So after some more thinking, I've decided that what I need to do is:

1) Open a portion of the binary file.
2) Convert that binary file into an array
3) Use the array to make a graph and a spectrogram
4) Close that portion of the binary file.
5) Open the next portion of the binary file and repeat until the end.

I can do 2,3, and 4, but I'm having trouble opening just a portion of
the binary file and then saving that to the program (so that I can
append the next portion of the binary file's data). Any suggestions?
Thanks.

Jennings




jyxmix@gmail.com wrote:
> Here's an update:
>
> I can open the file using the Read from Binary File function, but my
> computer says that it runs out of memory after I try to play parts of
> the file (I convert the binary file to a wave file and then play it).
>
> Any suggestions would be greatly appreciated.
>
> Jennings

[JLS]

Hello,

From your previous post, it sounds like you're able to open and read from the file - can you try reading half (the get file size function will be helpful) and immediately writing that half to another file?  If this works, you can try the same thing with the second half using the set file position function.  You could repeat this process until you had files which were of manageable size for your application/processing needs.

I hope this helps!

Best Regards,

JLS

[Anonymous]

Thanks for the reply,

That would kind of work, but ultimately, I need a graph that
contains the entire file. In other words, I need to convert the binary
file to a wave file that I can view as a graph. If I only open one
half, then I would only see the graph from that first half, which is
not useful to the project. I need to view the wave file in its
entirety as a waveform graph, so is there any way to add the data from
the second half of the file to the first half?

I guess it comes down to the memory buffer size, right? Labview tries
to load the entire file onto the RAM, but I need it to load a portion
onto the RAM, save that portion on the hard drive, load a second
portion onto the RAM, and add that portion to the first portion on the
hard drive, and so forth.

Is there any function in labview that can do this? Or can I only
manipulate data using the RAM, and not the hard drive?

Thanks for any and all help!

Jennings



JLS wrote:
> Hello,
>  
> From your previous post, it sounds like you're able to open and read from the file - can you try reading half (the get file size function will be helpful) and immediately writing that half to another file?  If this works, you can try the same thing with the second half using the set file position function.  You could repeat this process until you had files which were of manageable size for your application/processing needs.
>  
> I hope this helps!
>  
> Best Regards,
>  
> JLS

[JLS]

Hello,

I'm wondering if it even makes sense to try to view that much data - it won't take very long before you don't have enough pixels to represent it accurately on your screen.  Thus, I think it would make sense to break your data into smaller pieces, each of which could be viewed accurately on your screen, OR, sample your data (take every 10th or 100th point or something) - this is really all you'll be seeing on your screen anyway if you try to plot a huge data set.

Does that makes sense?  If you need to scroll through your data or something, perhaps there are some programming tricks we can play to make sure we keep memory usage down, and only have memory corrsponding to what we need to show in the graph, which I just argued was probably not all of your data.

Looking forward to your repost!

Best Regards,

JLS