transforming large data arrays

[DFGray]

LabVIEW 7.1 uses signed 32-bit pointers for file access, so directly specifying a location in a file will fail if the file is over 2GBytes in size.  However, you can still access it if you stream data from disk and do not specify the location.  See the fifth section of Managing Large Data Sets in LabVIEW for details.  The tutorial also includes sample code, although I don't think it is in LabVIEW 7.1.  I attached the 7.1 code below so you won't have to ask.

The tutorial will also give you some tips on how to speed performance.  For disk read/write, the most important one is chunk size.  For a Windows system, this should be about 65,000 bytes.  This is usually the rate limiting step.

Let us know if you need more help.

[charliedawkes]

Well after that the company has upgraded to labview 8 so it seems to work now! Thanks for you help

[eustis74]

Greetings,

  I have a somewhat similar problem, but I don't believe it can be solved in this manner.  I am trying to concatenate a bunch of very large SDF files (essentially text files).  I can do this via read from text, concatenate strings, and save.  However this only works for about 8 files since they are each over 120 mb and my system runs out of memory.  I have about 1000 of these to do.  I would like to do them all at once, but I can try to do 100 at a time, much less and I am kind of defeating the purpose. 

Can someone look at my code and give me some pointers?  My thought has been that I need to somehow pass the end of file to the concatenate function without loading it all to memory.  I have run out of ideas.

Many thanks.

[altenbach]

Read your files in a loop and append their content to the existing target file on disk. No need to concatenate in memory.

[altenbach]

here is a quick draft of what I had in mind (LabVIEW 2010).

See if it works for you.

(Your code was a bit convoluted, with hidden indicators serving as unneeded local variables. None of those are needed. Also the execution order is determined by the data dependency, so no need for a sequence structure. I also don't understand the need for the big case structure. Are you using continuous run?)