Undesired buffer allocation

[mishklyar]

Hello,

   May be it's possible somehow to avoid buffer allocation in the posted picture.

(I receive "out of memory" message at the buffer allocation )

It is LV 8.2, so there is no "in-place" structure.

 Thanks,

Michael

[andre.buurman@carya]

Don't use auto indexing.

Do you need the data from "Flush queue"?
Handle the data from "Flush queue" inside the for loop directly instead, minimize the data that exits the loop inside memory.
Let the data remain in the array of clusters containg the 2D array data until you want to process it.

[mishklyar]

Thanks for an answer,

   I just need  a 3D array.Some processing may be done on  2D slices,but at

the end, there is 3D FFT, so the whole 3D array needed.

Thanks.

[andre.buurman@carya]

Process data in smaller pieces and concatenate after processing.

[Ray.R]

Hi Andre,

What about getting rid of the Flush queue as you suggested by doing the appropriate producer / consumer loops?  That would mean changing the For Loop to a While Loop and de-queueing the elements within the 2nd While Loop.  If you enable the indexing out of the While Loop you'll end up with a 3D array.

See attached example.

RayR

[altenbach]

---

mishklyar wrote:

It is LV 8.2, so there is no "in-place" structure.

---

I don't think the "in place" structure would be of any help. IF you want that 3D array, you need to allocate memory for it, one way or another!

If you get "out of memory" error, your problems are probably a little deeper than that. Can you tell us a little more about your application.

How big are the queue elements? How many elements do you accumulate? Are the 2D chunks always of the same size?

It seems to me that you are abusing a queue as a plain accumulator of infinite size. This cannot be cheap! You are allocating huge amounts of complicated data structures inside the queue and then you need to allocate again for the 3D array.

Do you know the final number of slices? In this case you could allocate the final 3D array once in a shift register and replace with the new slices inside the loop directly. No queue needed.

See, the queue carries a lot of extra baggage, because it needs to allow for an "infinite" amount of elements of possibly different size each, which complicates the internal data structures significantly and cannot be very efficient.

If the 2D arrays are always the same size and the number of slices is also known, the allocation could be done much more efficient if you do it yourself.

In your current design, you need to allocate the cluster of queue elements and the 3D array (in addition to the allocation inside the queue). Since they are structured differently, the conversion cannot be done "in place" anyway, so don't get you hopes up. If you want to avoid the allocation of the cluster array, you could flush with a FOR loop as follows. I am not sure if it would provide much relief, though. I think you need to redesign your program from the ground up. :).

If you want, attach your code and provide information in the dimension sizes.

Message Edited by altenbach on 06-29-2008 09:30 AM

[mishklyar]

Hi altenbach,

The application is MRI data aquisition, 2D array is X-Z slice, and third dimension is Y.

In general the goal is to get images as big as possible.I started to get memory error messages at 128*256*256.

The data is complex.Yes, all slices are the same size,all sizes known before the application starts.

I attach here simplified VI.

   Thanks, Michael.

[Ray.R]

I just "popped in" and probably don't understand something.

However, one thing I didn't indertand is why the 2nd queue was needed.  As I mentionned in my previous post, the while loop of the consumenr loop can create the 3D array without the additional queue.

I attach your code modified without the 2nd queue.

RayR

[mishklyar]

Hi JoeLabView ,

I run your VI,and look at the size of the 3D array at the end.Every time it was different.

However  I understand your idea.Just need to decide tomorrow wich way is more efficient:

  1) Of preallocated 3D array, and substituting  2D arrays at apropriate index.

  2) Your way, with carefull timing and/or stop condition.

Buy the way,may be it is better to realize the queue after the consumer loop at your VI?

Thanks,

Michael

[altenbach]

OK, way too complicated!!!!

You know exactly how many elements will be in the queue, so you can just use a FOR loop in the lower part.

We can wait with an infinite timeout and we are done when 32 slices have arrived. guranteed!

Here's a quick draft.

I assume that the only reason you are using a queue is that acquisition is slow and you want to start processing before waiting for all slices.

Is this really worth it?

How long does the aqcuisition take and how long does the processing take?

(A) If the processing is very slow compared to the acquisition, the queue is not really needed and you could process everything later.

(B) If the processing is very fast compared to the acquisition, you can do all at the same time in one loop.

Message Edited by altenbach on 06-29-2008 02:23 PM