LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

2D array memory allocation issue

Hi

 

I have a 3d array, named inputs, with 10000 pages. Each page is a 2d array with 3 rows and 3 columns. 

I have another 2d array named indices, with 13732 rows and 3 columns.

 

I am trying to replace the indices in the Initialized array with the values in the input array.

 

Initially I was trying to replace the indices in the Initialized array as soon as each page of the 3d array inputs is created. This lead to the memory full error in Labview. I tried the in place elements for replacing the array elements, tried changing the double precision input to single precision, but it did not give me any improvement. 

 

So I tried to take the 3d array separately and then tried to replace the indices of the 2d Initialized array. This seems to work, but I am not sure if this is a solution, or is there a better way to deal with such large arrays.

 

I would appreciate some help to solve this issue. A VI with all necessary data and my try is attached.

0 Kudos
Message 1 of 13
(2,988 Views)
  • You cannot replace indices, just values at given indices.
  • Not sure why you call it "reshape" in the diagram comment. It is not.
  • Most of your "input tunnel followed by index array" can be replace by autoindexing.
  • Are you using LabVIEW 32bit or 64bit?

Here's arguably simpler code that does exactly the same:

 

 

altenbach_0-1615166143157.png

 

0 Kudos
Message 2 of 13
(2,933 Views)

Thanks for the reply and apologies for messing up with the question. It was not a reshape of the array.

 

I am using 32 bit Labview. The memory used for this operation seems to be high, around 1529.29 Mb (screenshot attached). I read about the Data Value References. Will they be of any help?

0 Kudos
Message 3 of 13
(2,911 Views)

I'm not quite sure I understand what you want to do.  Here is what I think is the situation:

  • You start with a 3D array having 13731 3x3 Arrays "inside" it (13731 is the first index, the other two are 3x3).
  • You basically want to operate with the 3x3 Arrays, either specifying the first index sequentially or possibly "at random", but you basically want to extract one or more 3x3 Arrays.  [Note this is the part I'm least certain about).
  • You might (or might not) also want to modify/replace some of the 3x3 sub-Arrays.

This might be a perfect place to use a Data Value Reference.  You need enough memory to allocate a single copy of the 13731 x 3 x 3 Array, create a New Data Value Reference to it, and can then "point to" any sub-Array within this and (without having to create another copy of the Array) can directly read these 3 x 3 = 9 values, requiring only another 3 x 3 entries.  

 

Bob Schor

Message 4 of 13
(2,892 Views)

Hi

 

Thanks for your response. Let me make it a bit more clear.

 

Let me assume that the index 0 of the Indices array is 2706, 2734 and 3582.

 

The index 0 of the 3d input array is a 2d array with 3 rows and 3 columns and is shown in the attached image, pic1.

 

So the result I want to have is as follows

 

1. The element at index 2706,2706 of the Initialized array, must be replaced by element 1,1 of the input array + element already present in 2706,2706.

2. The element at index 2706,2734 of the Initialized array, must be replaced by element 1,2 of the input array + element already present in  2706,2734.

3. The element at index 2706,3582 of the Initialized array, must be replaced by element 1,3 of the input array  + element already present in 2706,3582.

 

And so on for the first page of the 3d array.

 

This has to be continued for all rows of the indices array. I tried using the New Data Value Reference, and my try is attached below. Since I am doing it for the first time, I am not sure whether I am doing it the right way.

 

On running the Profile performance and memory, I get that the average bytes is 1148.41M. I suppose its a very high value as I may have to work with an indices array which might be 50000 x 50000. 

 

Thanks for your time and help.

Download All
0 Kudos
Message 5 of 13
(2,887 Views)

The resulting array is 380,880,000 Bytes. (6900X6900X8).

 

The control will contain a copy, (800MB) Maybe there's a 3rd copy (1200 MB)

.

During editing, there will be copies of the data stored in the undo. So even more memory is needed,

 

A DVR might help to prevent copies (that shouldn't be there). But the way you use the DVR now seems pointless. It might make things worse. Sometimes LV does make copies that are hard to explain. If you start passing this data through multiple subVIs, a DVR might help.

 

I'd use an in place element structure for the replacement:

Memory Allocation.png

 

Message 6 of 13
(2,853 Views)

A quick 'fix' is to switch from doubles to singles. If you can get away with it.

Message 7 of 13
(2,849 Views)

On a side note, if you try to compare the results of various methods, you will never get an equal, because your array contains thousands of Inf and -Inf each and there are over a hundred cases where you get a (-Inf)+(+Inf), which will turn into a NaN in the output.

 

If this is a concern, you could filter the data in the innermost loop.

 

Is this code to be used flat or turned into a subVI? How often does it get called? I agree that the use of a DVR is not needed unless the output data is needed is vastly different places and times.

 

Where is the data needed? How is it used later? Your output array is 99.9% zeroes, which is NOT interesting! The interesting data is only 0.1% of that gigantic 2D array. You could get away with 1000x less memory use by utilizing more efficient data structures than a continuous memory block. For example, it would be easy to use a map with an array of two indices as key and the value as data. See picture and attached code. Once you have the MAP (using a tiny fraction of the memory compared to the 2D array, you can access any element by index including the ones that are zero, i.e. don't exist in the map.

 

(LabVIEW also has a sparse matrix toolkit, but I am not sure if that would be useful here.)

 

altenbach_0-1615222675460.png

 

 

Message 8 of 13
(2,834 Views)

Here's how to do the map solution and how to later retrieve values. (~1000x less memory usage!!!)

 

altenbach_1-1615223382116.png

 

Message 9 of 13
(2,831 Views)

Hi

 

Many thanks for your response. I am trying your solution and in the mean time to To answer some of your queries

 

Is this code to be used flat or turned into a subVI? How often does it get called? 

 

This is a Sub VI and it will be called only once.

 

Where is the data needed? How is it used later?

 

The data will be needed in a following Sub VI where I plan to take the inverse of the large 2d array.

 

So I am interested to get the whole 2d array for processing at a later stage in another sub VI.

 

Thanks

0 Kudos
Message 10 of 13
(2,814 Views)