How to process huge files?

[maksya]

Hi, dear ALL

I have a file which size is about 200 Mbytes (and greater sizes are possible). It contains some kind of log information. And I want to read this file in LV and process the log.

There is a problem - I read a file via Read From Text File.vi with option "-1" (entire file) - case1.vi attached. The VI is hanged =( LabVIEW eats all the PC's RAM while reading and exit with error with description like "Not enough memory to complete this operation"..

All right, then. I have decided to read the file part by part - case2.vi attached. There is no hangs but reading process is too slow - for about 17 Mbytes in 5 minutes.

What is the proper magic to operate with huge files in the LV. Please help with advices.

Regards,

Max

[RavensFan]

What are you trying to do with the files?  You say you want to "process" them, but you aren't doing anything but reading them in and displaying in an indicator.

Case 2 is actually worse, because although you are reading it in, you are still concatenating it together into one large block of data.  But since you are doing it in parts, every time you had another part, the memory manager has to go out and look for a new, larger, and contiguous block of memory to store it in.

When you process the file in parts, you actually want to do something with the data in parts, discard the old parts and move on to the next part.

See Managing Large Data Sets in LabVIEW.

Message Edited by Ravens Fan on 04-30-2010 12:22 PM