03-07-2021 04:55 PM - edited 03-07-2021 05:25 PM
Hi
I have a 3d array, named inputs, with 10000 pages. Each page is a 2d array with 3 rows and 3 columns.
I have another 2d array named indices, with 13732 rows and 3 columns.
I am trying to replace the indices in the Initialized array with the values in the input array.
Initially I was trying to replace the indices in the Initialized array as soon as each page of the 3d array inputs is created. This lead to the memory full error in Labview. I tried the in place elements for replacing the array elements, tried changing the double precision input to single precision, but it did not give me any improvement.
So I tried to take the 3d array separately and then tried to replace the indices of the 2d Initialized array. This seems to work, but I am not sure if this is a solution, or is there a better way to deal with such large arrays.
I would appreciate some help to solve this issue. A VI with all necessary data and my try is attached.
03-07-2021 07:16 PM
Here's arguably simpler code that does exactly the same:
03-07-2021 08:00 PM
Thanks for the reply and apologies for messing up with the question. It was not a reshape of the array.
I am using 32 bit Labview. The memory used for this operation seems to be high, around 1529.29 Mb (screenshot attached). I read about the Data Value References. Will they be of any help?
03-07-2021 09:39 PM
I'm not quite sure I understand what you want to do. Here is what I think is the situation:
This might be a perfect place to use a Data Value Reference. You need enough memory to allocate a single copy of the 13731 x 3 x 3 Array, create a New Data Value Reference to it, and can then "point to" any sub-Array within this and (without having to create another copy of the Array) can directly read these 3 x 3 = 9 values, requiring only another 3 x 3 entries.
Bob Schor
03-07-2021 10:20 PM - edited 03-07-2021 10:46 PM
Hi
Thanks for your response. Let me make it a bit more clear.
Let me assume that the index 0 of the Indices array is 2706, 2734 and 3582.
The index 0 of the 3d input array is a 2d array with 3 rows and 3 columns and is shown in the attached image, pic1.
So the result I want to have is as follows
1. The element at index 2706,2706 of the Initialized array, must be replaced by element 1,1 of the input array + element already present in 2706,2706.
2. The element at index 2706,2734 of the Initialized array, must be replaced by element 1,2 of the input array + element already present in 2706,2734.
3. The element at index 2706,3582 of the Initialized array, must be replaced by element 1,3 of the input array + element already present in 2706,3582.
And so on for the first page of the 3d array.
This has to be continued for all rows of the indices array. I tried using the New Data Value Reference, and my try is attached below. Since I am doing it for the first time, I am not sure whether I am doing it the right way.
On running the Profile performance and memory, I get that the average bytes is 1148.41M. I suppose its a very high value as I may have to work with an indices array which might be 50000 x 50000.
Thanks for your time and help.
03-08-2021 04:39 AM
The resulting array is 380,880,000 Bytes. (6900X6900X8).
The control will contain a copy, (800MB) Maybe there's a 3rd copy (1200 MB)
.
During editing, there will be copies of the data stored in the undo. So even more memory is needed,
A DVR might help to prevent copies (that shouldn't be there). But the way you use the DVR now seems pointless. It might make things worse. Sometimes LV does make copies that are hard to explain. If you start passing this data through multiple subVIs, a DVR might help.
I'd use an in place element structure for the replacement:
03-08-2021 04:55 AM
A quick 'fix' is to switch from doubles to singles. If you can get away with it.
03-08-2021 11:00 AM
On a side note, if you try to compare the results of various methods, you will never get an equal, because your array contains thousands of Inf and -Inf each and there are over a hundred cases where you get a (-Inf)+(+Inf), which will turn into a NaN in the output.
If this is a concern, you could filter the data in the innermost loop.
Is this code to be used flat or turned into a subVI? How often does it get called? I agree that the use of a DVR is not needed unless the output data is needed is vastly different places and times.
Where is the data needed? How is it used later? Your output array is 99.9% zeroes, which is NOT interesting! The interesting data is only 0.1% of that gigantic 2D array. You could get away with 1000x less memory use by utilizing more efficient data structures than a continuous memory block. For example, it would be easy to use a map with an array of two indices as key and the value as data. See picture and attached code. Once you have the MAP (using a tiny fraction of the memory compared to the 2D array, you can access any element by index including the ones that are zero, i.e. don't exist in the map.
(LabVIEW also has a sparse matrix toolkit, but I am not sure if that would be useful here.)
03-08-2021 11:10 AM - edited 03-08-2021 12:53 PM
Here's how to do the map solution and how to later retrieve values. (~1000x less memory usage!!!)
03-08-2021 04:28 PM
Hi
Many thanks for your response. I am trying your solution and in the mean time to To answer some of your queries
Is this code to be used flat or turned into a subVI? How often does it get called?
This is a Sub VI and it will be called only once.
Where is the data needed? How is it used later?
The data will be needed in a following Sub VI where I plan to take the inverse of the large 2d array.
So I am interested to get the whole 2d array for processing at a later stage in another sub VI.
Thanks