LabVIEW
Extremely large files
Ton,
Ah, you must use an old LabVIEW version then. In newer version, these are atomic functions that cannot be inspected.
Still, I doubt they are inefficient. 😉
Camu
Anyway, using 20M input has a memory footprint of over 300MB. Since arrays need to be contiguous in memory, you can run into problems. You need to be very careful, because even a single extra data copy (local variable, poor coding) can push you over the limit. Of course here we double the problem because we compare the direct with the file version. In real life, you need only one of the two, cutting the memory footprint in half.
There is no problem with indexing arrays of any size (size is I32) as long as we don't exceed the memory capabilities.
Well, the filter VI creates 3 arrays of the output size, even though you only need one. I would modify this or even flatten it out to the main VI.
A lot of your code is duplicated and could be consilidated in a FOR loop looping twice and reusing the buffers.
You create way too many arrays in memory. (you create e.g. two boolean arrays with 20M elements and 6 filtered arrays before processing the histogram. Also many operations don't need to be done on arrays. For example you have an array and then subtract one constant and then another constant. It is sufficient to add the two constants first and then subtract the result from the array. Same with the two divisions in the loops.
What you should do is generate the two histograms right there from the raw data. Here's a quick draft how you could do it (LabVIEW 8.2).
(how many points are NOT 1 or 2? If this number is small, you can take the scaling out of the case structure).
Please verify correct operations and modify as needed.
You can easily see where new memory buffers are created. Simply go to "tools...profile...show buffer allocations" and look for the black dots on the diagram. 😉
For the subtraction, the memory of the input array is re-used (not released and newly allocated), so memory wise, this is not a problem. However, doing both subtraction on the array will need 2N operations, while adding the two scalar values first and then subtracting from the array is only N+1 operations. 50% less work!.
Camu wrote:
Oh, I don't know that labview will not release the memory when one operation is done, e.g. after substracting a constant from an array (I thought the memory will be released). Is it because the two boolean arrays and 6 filtered arrays are used by the same subvi--filter.vi, that labview will store all 2+6 arrays in the memory, waiting to be called by filter.vi.
--------------------------------------------------------------
(how many points are NOT 1 or 2? If this number is small, you can take the scaling out of the case structure).
------------------------------------------------------------------
BTW, a stupid question, what do you mean "take the scaling out of the case structure"? 
The subtraction, division, and rounding code is the same in case 1 and 2, but missing in the default case. You can avoid the duplicate code by doing this calculation before the case structure (or even before the FOR loop, at the cost of another array buffer allocation). Of course whenever the default case is executed (which does not need the scaled data), you did all the scaling calculations for no reason. Typically all code that is the same in all cases, belongs outside the case structure. 🙂
Here are the two alternatives. I would recommend to do some benchmarks to see what's better.
I would also recommend to use the "in place elements" structure to built the histogram (see my earlier example).
Message Edited by altenbach on 06-23-2008 04:16 PM
