LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Queue Memory Allocation Weirdness

I am at a loss to how LabVIEW is allocating memory in its queues.

 

In the example I attached you must open Windows Task Manager and look at the Memory column.

 

I'll try to lead you through the steps of the example.

 

1.) Make a queue of length 10, data type double array.

 

2.) Fill the queue with 10 arrays of 1M points, watch the memory increase in 8M steps in the Task Manager. (This makes complete sense to me.)

 

3.) Empty the queue of the 10 elements. Note that the memory stays constant in the Task Manager. (This also make sense to me and LabVIEW will not release the memory in case it is needed later.)

 

4.) Fill the queue with empty arrays, watch the memory decrease in 8M steps in the Task Manager. (This also makes complete sense to me.)

 

5.) Put one array element in the queue, 1M doubles, then dequeue the element. Now the memory increases by 8M steps for all 10 iterations. Note if you increase the number of iteration on this last loop, the total memory used never goes beyond 80M, the original allocation.

 

Why is LabVIEW using all the memory initially allocated to the queue when there is only 1 element in it at any time?

 

I ask this because at points in my real program I want to limit my queue to a single element to save memory, but this example shows that will require a work around.

 

If anybody has any insights, I greatly appreciate it.

 

Cheers,

Andrew

Download All
0 Kudos
Message 1 of 14
(5,615 Views)

I'm not going to claim to have the exact details, but I'm pretty sure the weirdness is coming from the fact that you are using arrays in the queue.  As each array is "replaced" in the queue, LabVIEW deallocates the memory for the old array and reallocates for the new.  I'm guessing it will use what memory is there and allocate/deallocate as needed.  Maybe someone from NI can elaborate a little better than I.


GCentral
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
"Not that we are sufficient in ourselves to claim anything as coming from us, but our sufficiency is from God" - 2 Corinthians 3:5
0 Kudos
Message 2 of 14
(5,609 Views)

Thanks.

 

But I am still unclear. After inserting and removing an array that memory space should be available, why is LabVIEW using a new memory space when there are no elements in the queue?

 

thanks again.

 

Cheers,

mcduff

0 Kudos
Message 3 of 14
(5,604 Views)

I'm still working on what's going on here, but in case it helps: if you do a single enqueue outside the last for loop, the inside the loop do a dequeue-enqueue pair where the dequeued element is immediately enqueued, you will get only the memory use for a single element.  What puzzles me is that if you leave the enqueue/dequeue as is, but put the dequeued element in a shift register and enqueue that, you get the same behavior you're seeing.


mcduff wrote: 

I ask this because at points in my real program I want to limit my queue to a single element to save memory, but this example shows that will require a work around.


Single-element queues are generally efficient.  Do you plan to have a 10-element queue with only one element in it, or you're going to limit the queue to a single element when you first obtain it?  I'm not certain any workaround is necessary depending on how you plan to use it, based on my above results with doing the dequeue/enqueue in the opposite order within the loop.

0 Kudos
Message 4 of 14
(5,592 Views)

Nathand,

 

Thanks for your advice, although I am having a hard time following it. I'm trying to put the words into images in my head, but presently it is not working. It has been a long week. 🙂

 

As far a real program goes, a brief description is as follows:

 

I have a bunch of A/D boards in a PXI chassis; the slow part of my program is writing to a file. After the data is processed, I store the values in a queue where it is passed to a "consumer loop" that saves the data. If I have a single element queue, then my DAQ acquisition speed is limited by how fast the files can be saved. Thus, I have a finite length queue to buffer the data while it gets saved.

 

The person that I work likes to store millions of double precision points, so memory can grow fast. Typically we only get 2M total points from each board, and my queue can grow to 500MB in size. However, there are times when my colleague wants to acquire 10M or more points from each board. In this case, I want to limit the queue size to 1. Since this can not be done dynamically while the program is running, I am using a semaphore like construct to limit my queue to 1 element. (That is I wait until my file is saved before adding another element.) When I tried to do this, my memory usuage kept growing, similar to the example I provided earlier.

 

I came up with a kludge type solution, fill up the queue to N-1 elements with empty arrays, then insert in the front my real data set. It works but I am not happy with it.

 

On this board there are plently of people like yourself who are much more clever than me, and i am hoping they can figure out what is going on here. (I'm still trying to figure out your advice.)

 

Thanks again for your help.

 

Cheers,

mcduff

0 Kudos
Message 5 of 14
(5,586 Views)

My comments were more observations than advice.  Here's the modification I made where you don't see an increase in memory (I also shortened the wait times in most of the loops).  I'll be curious to see what others contribute to this discussion as I don't fully understand what you're seeing either.

Queue Memory Allocation Weirdness.png

Message 6 of 14
(5,582 Views)

Thanks for the picture!

 

Have a good weekend!

 

mcduff

0 Kudos
Message 7 of 14
(5,580 Views)

A couple of more observations. (I am hoping some of the knights of NI pick up this thread.)

 

1.) If the array size is increased from the orginal size, for example 1M to 2M points,  in the last loop of the example I posted, then the memory increases to Number of elements in queue * array size. That is, even though the queue only has 1 element in it, the memory allocation acts like N elements.

 

2.) Another way around this is to enqueue DVRs of arrays; here the memory acts as expected; although I have the feeling it is a little slower since memory is being allocated and reallocated.

 

Although Nathand's solution works, I do not understand why, and unfortunately I cannot incorporate it into my VIs as I cannot do anything outside of a loop. (I have a producer consumer type loop.)

 

Cheers,

Andrew

0 Kudos
Message 8 of 14
(5,550 Views)

@nathand wrote:

My comments were more observations than advice.  Here's the modification I made where you don't see an increase in memory (I also shortened the wait times in most of the loops).  I'll be curious to see what others contribute to this discussion as I don't fully understand what you're seeing either.

Queue Memory Allocation Weirdness.png


 

Not on a modern LV machine at the moment...

 

What happens if you change the array size constant to a control?

 

What about an "always copy" on both wire branches to rule LV trying to use the same buffers in both places and latter decide it has to duplicate after all.

 

How about some "show buffer allocations" dots, what do they show?

 

Ben

 

Retired Senior Automation Systems Architect with Data Science Automation LabVIEW Champion Knight of NI and Prepper LinkedIn Profile YouTube Channel
0 Kudos
Message 9 of 14
(5,546 Views)

Ben,

 

Thanks for your insights. However, please try this example while looking at the task manager in Windows.

 

QueueMemoryWeirdness2.png

 

 I am using LabVIEW 2011 SP1 and Win7 with a Core2Duo.

 

The buffer allocations do not show anything unusal or unexpected. The only strange part is the last loop; the memory keeps increasing even though there is only 1 element in the queue at anytime.

 

Cheers,

mcduff

0 Kudos
Message 10 of 14
(5,543 Views)