, Memory wasting

[IngoWolf]

Hi group,

because of out of memory errors i played with the profiler to see the memory usage and had to found out, that my labview 8.5 uses much more memory than the size of the data I deal with. I read out 8.000.000 samples of I/Q Single values 2x32 Bit which is 64MBytes, Labview needs 350MB to read it out and save it to a 2D array, the driver VI needs 150MB of which is not to much. If I compute the amplitude of the IQ-Data using 2x index-array to get I and Q-Array, 2x x^2 + 1 x plus and wire the result (32MB) to an chart after, the programm needs 570MB overall, why additional 220MB are needed for this operation. If I wan't to do all with 16.000.000 samples I usualy run out of memory while 1,2GB is still well below of the 2GB RAM I have, the /PAE switch also seems not to make things any better (is shurly recognized)  and the /3GB makes my XP Laptop not working allright.

The operations are done on Buttonclicks in a messagestruct in a while loop no tunnels are needed. I allready tryed the optimizations described in the handling large data papers - was worser before - but I think not doing fundamental errors any more. I can't understand that math of memory using.

[RavensFan]

I would recommend posting your VI.

Turn on coercion dots, that may show a change in datatype that would cause extra memory to be used.  Do you have any branches on your main data wire?

There are several experts on the forum who are excellent had determining where memory is being used.  I wouldn't be surprised if Altenbach or TST jump into this thread.

[altenbach]

I agree with Ravens, post your VI. 🙂

---

IngoWolf wrote:

...and wire the result (32MB) to an chart

---

Why a chart??? What is the history lenght?

---

IngoWolf wrote:

The operations are done on Buttonclicks in a messagestruct in a while loop no tunnels are needed.

Sorry, I am not familiar with that terminology? If you don't have tunnels, how is the data communicated between event cases? Do you use local variables?

[IngoWolf]

OK here is a part of the VI consuming >200MB memory, transfering 8.000.0000 IQ SGL Samples = 8.000.000*2*4 = 64 MBytes

The resulting Amplitude wired to the vector should have 32MBytes

[GerdW]

Hi Ingo,

ok, you have:
- "IQ-Data" storing 2×8M SGL values
- "IQ-Betrag" storing 1×8M SGL values
- you create intermediate buffers in the "index array": 2×8M SGL values
- your local "IQ-Data" is also storing 2×8M SGL values

Just this makes up: 7×8M SGL values= 7*4*8 MB=224MB...

Try to:
-avoid local variables, especially on big arrays
-use in-place operations when possible
-split up big arrays into smaller portions for the processing
-decimate big arrays when displaing in charts/graphs as it's senseless to show 8M pixel in a graph less than 400 pixels wide...

Also:
-don't use the word "chart" when you use a graph! (At least not in the label, you may use the caption to show this name to the user!)
-it's not "nice" to attach vis with auto-run enabled (kind of viral behaviour )

What did you do with your "IQ-Data" indicator? When I un-hide the indicator an array was showing up with apparently no numeric element in it... I couldn't select any array element?!?


Message Edited by GerdW on 02-27-2008 08:37 PM

[altenbach]

OK, your local variable contains an extra data copy and is not needed. You can turn your array into an indicator and hook it up directly.

If you show buffer allocations, you have 64MB in the indicator, 64MB in the local variable, and 2x 32MB (in each of the slices). ~200MB as expected.

Please consider to represent your 2D data as a 1D array of CSG (complex single), that seems to be a more natural format for the data and keeps the code simple. In your case, you could just use the sqaure of the absolute value. (See bottom image)

Now you have 64MB in the control and 32MB in the SGL array after taking the absolute value. ~100MB, or half of your version. 😄

Please don't:

1. Set the VI to run when openend.
2. Hide controls, and array elements within it.
3. Don't call it a chart if it actually is a graph. 😉

Message Edited by altenbach on 02-27-2008 11:41 AM

[altenbach]

---

GerdW wrote:
What did you do with your "IQ-Data" indicator? When I un-hide the indicator an array was showing up with apparently no numeric element in it... I couldn't select any array element?!?

---

You need to right-click inside the array container and select "avanced..show array element".

On top of that, the array contianer and the elements are both set to disabled&greyed"

Currently, the array elements are hidden inside a hidden indicator. 😮   Why!!!!!????

Message Edited by altenbach on 02-27-2008 11:52 AM

[GerdW]

Hi Christian,

never made such wierd things on my own...To hide elements in a hidden array
But again I could learn something useful (?) by participating in the forum

[altenbach]

Just for illustration, here are a few alternatives to do the same thing. Let's assume that you input array has 2 rows and n columns for a total size of x in memory.

(The black dots show buffer allocations).

1. Your code fragment (without the local). Memory use is 2x.
2. The same with a single resized index node. Cleaner diagram, but still 2x.
3. Looping over each column is more efficient. Memory use is 1.5x+2 (+peanuts).
4. If your array were transposed, the code in (3) could be made cleaner. It is important to decide on the most suitable data arrangement. Since we need to transpose, memory is 2.5x+2. If the array were transposed from the beginning, memory use would be the same as (3), but might run a bit more efficient because it operates on adjacent memory locations.
5. If you would represent your data in complex, the memory use is 1.5x.

Message Edited by altenbach on 02-27-2008 01:02 PM

[IngoWolf]

Hi group,

IQ-Data is filled up in other parts of the code (other event cases), its the global container for my IQ-Data.

I have to use local variables to work with in my event cases.

The >200MB is only an addition (570MB overall)

OK the local variable makes 64MB the split 64MB more but doesn't Labview deallocate the copies from the

beginning when dealing with the split to make the amplitude and deallocate the split copies before copiing the

Amplitude to the Graph? The other event-Cases also only work with the IQ-Data-Memory in similar

ways. Isn't the temporary memory freed when the event case is completely processed so it  can be used

when other event cases are processed?

In my opinion the code in the event case should use 128MB (2x64MB) at max because the memory

from the earlier steps can be reused (deallocated)  in the later ones.

And the other event cases should be fine with the same 128MB doing similar things.

I need all the data in the graph because the user should be able to zoom in up to one sample, and its

only 32MB.

Labview shows me where buffers are allocated but where are they deallocated?

Ingo