Can I read long files (Gigabytes)?

[Mr.farmcorp]

Hi, I hope you can help me.

Well, I want to read a file but this file will be long, order of gigabytes. My PC has 1024 MB RAM, Dual Core Intel 1.73 GHz, and Windows Seven. I have been searching in this site and  I found some information about it and then I developed the next points

1.- I created a call reference VI

2.- I use cylce to call my VI that read the file, I programmed it to read for parts. In the cycle I use a call reference VI

3. - I use cylce to call my VI that read the file buts in this test I use a simple subVI

Sometimes I received a message "Error, not enough memory  to continue",  or just my pc gets stuck.

Thank you.

[Darin.K]

1 Gig RAM and Win7, how do you even open LV without grinding the Hard Drive?  You are probably using virtual memory from the moment you start.

[EricC.]

Hello

When you use long file, you cannot read all the file.

You must read only a part of the file

You can use binary acces or datalog access. The idea is the same.

when you read a data (here by binary access), the file pointermove to the next position. So you don't have to manage the current position.

If you close you file after reading, you must save the current position.

By this way you don't load all the file.

[altenbach]

Your points 1-3 make very little sense without also showing some actual VIs.

Of course you cannot read the entire file into memory, but I assume you have decided on reasonable chunks. What is the datatype in memory? Remember that numeric arrays need contiguous memory and correct design is important so that consecutive reads re-use the same buffer. Also avaid extra data copies in memory (avoid locals and extra indicators and keep the front panel of subVIs closed. Avoid all code elements that force the front panel of subVIs to be in memory. Show us your code and provide more information.

What is the format of the file? (binary, formatted ASCII, datalog, etc.)

What is the datatype in memory? (string, DBL, U8, etc)

[lfelias]

Hi, Mr.farmcorp

What I think you can be able to do is to read chunks of data big enough to work with, in the count terminal you are able to change the amout of data to read.

So, you can get maybe 20 MB then work with them, erase that from memory and then read another 20 MB.

The 20 is just a guess, it might be a larger amount of data or smaller chunks, i don´t know.

Good luck and let me know how that goes!!!

[Mr.farmcorp]

hi, thank you for aswer.

here you are my VIs.

Fast description:

Vi called: call by reference cycle, I use it to set the file size, location file, upper and lower limit (number of bytes) and indicators. This VI makes a call to subVI, but It is a call by reference.

VI called: call subVI, cycle; I do the same as described above, but It is a normal call to subVI.

VI called: subVI, is where I search my headers 129, 130, 131, 132. If the program find 129 then I have a data of time and for the rest of headers I have amplituds of some signal. For time (129) I need more cells to display: header, day, hour, second, etc. and for the rest (130, 131, 132) I just need two cells (header and amplitud).

there is another subVI but this only makes a decoding function and returns I32 data type (this is for cases: 130, 131, 132).

data type in the file is binary coded and the output will always be array of unsigned byte (U8).

[Mr.farmcorp]

hi Darin k.

I know tha my PC is limited but I have run other programs in LabView, I have even read small files (50 MB). My computer is limite but I'm sure that I can read in some mode to reuse memory and ignore my memory limiting. Please if you have some help, thank you.

[Mr.farmcorp]

hi Eric C.

Please check the VIs tha I send. In general: I open, read, show, close, repeat N times. So after each cycle I have an array of unsigned bytes and  Index (i of cycle), so I will join the sub arrays generated in each cycle.

If I undestand correctly, you say me that I should : open, read, show, repeat N times, close. 

Is this correct?, what could be the difference?

thank you.

[Mr.farmcorp]

hi ifelilas,

thank you for answer. In fact I have been using the terminal "count" and I read for parts. The problem is that I can't read all my file, because whe I try to read for example 200 MB (this is 10 cycles of 20 MB), LabView say "not enough memory" or my PC get stuck.

any other ideas?

this question is for all.

how about if I apply delays between each cycle?  Can I free memory???????

[Mr.farmcorp]

hi altenbach

thank you for your answer.