Problems for processing (matrice multiplication) large arrays

[TgTgTg]

Hello people,

I'm developping a (sub)VI which should generate the different Hadamard basis matrix. As input, it takes the order, the indexes of the basis, and other stuffs relative to the usage in my case, a correction matrix which values will replace the -1s in the basis and a Grayscale value.

Basically, I generate the complete Hadamard matrix using the function "create special matrix", a 2D matrix of size (order x order), then I transform it to a vector A of size (order^2, 1). And after compute the matrix R = A * transpose(A), which have a size of (order^2, order^2). All my hadamard basis are subarrays of size (order, order) in the matrix R. The purpose of double loops labeled "Generate the correct H(e,f) is to pick the correct subarray.

Now my problem is that, for order value equal or less than 128, everything works fine. From order = 256, it crashes when it should compute R (as shown in the PNG file). I don't now why !!!! Is it a memory problem links to the way matrix multiplication is implemented ?

Thanks for your help !!!

[Artem.SPb]

---

@TgTgTg wrote:

, which have a size of (order^2, order^2).

---

In the end, you want to get the matrix
256^2x256^2
or a total of 4294967296 elements.
If I'm not mistaken, this is 16GB of data.
Since this is a matrix; this amount of data should be allocated in one fragment. Even if you have more RAM, it may not have one such free segment.

[TgTgTg]

Thank you for your answer !

You're rigth !!!

I thought about that, but the problem is that this matrix is created by a multiplication of a row vector by a column vector.

Please do you have an idea on how I can allocate that memory and "tell" the multiplication function to use that memory ?

Thank you !

[Artem.SPb]

I have a similar problem: in my task I receive data by lines and need store long buffer of such data.
I've solve this task with DVR. But in this case you need your own handler.
Also you need LV 64bit.
Here simple example, but in LV 32bit I've reach i=6793 only ( I can't test it in 64bit now)
And in worst case you will store this data in file and read piece by piece.

And one more question: do you really need this huge piece of data? Try to find another way.

[TgTgTg]

ok ok !!! I will implement this solution, and tell you if it's works for me also.

I will also try to see if I can find another way to generate the basis....

[altenbach]

Of course the memory use is correct but your code is extremely convoluted. The upper part is basically just a reshape followed by an outer product (see image below). Since the values can only be 1 or -1, Converting to I8 would use 8x less memory (but the outer product would need to be done using a FOR loop. No big deal, right?) Also, the lower loop could use autoindexing of the output.

Sorry, I don't understand the algorithm. What are typical values/sizes for the "correct" array control? All the other controls? Why do you need that gigantic order^2 x order^2 matrix? Unless e and f are gigantic, most values are never touched again. (matrix indices each go from 0 ... order*order-1, but all you ever index outs is from 0 ... order+e-1 (or order+f-1, resp.)

Do you have a link to a website that explain the logic and algorithm? It does not make a lot of sense.

[altenbach]

In any case, you don't need to calculate that gigantic matrix at all. Given the two indices in the lower loop you can calculate the needed single value of the outer product right there in-place from the reshaped 1D array and the two indices. Keep it simple!

[TgTgTg]

Here is the algorithm, if you have matlab you can try it with NN=4, e and f values in {1,5,9,13}:

function g = hadamard2D(NN, e, f)
              a=hadamard(NN);
              [m, n]=size(a);
             for i=1:m
                  p(1+(i-1)*NN : i*NN,1) = a(:,i);
              end
              q=p';
              r=p*q;
             g = zeros(m,n);

             for o=1:m
                  for q=1:n
                        g(o,q)=r(e+o-1,f+q-1);
                  end
              end

end

- Yes, I can convert the values to I8, I will do it. Wow your propositions for optimizations are awesome. Thank you @altenbach

- For the lower loop, i'm not auto indexing because i just need a subarray in the (order^2 x order^2) matrix.

Actually, the chalenge I'm working on is to find a way to compute directely the basis I need without computing the (order^2 x order^2) matrix which contans all the basis (order x order) basis each having (order x order) elements.

[altenbach · Accepted solution]

---

@altenbach wrote:

In any case, you don't need to calculate that gigantic matrix at all. Given the two indices in the lower loop you can calculate the needed single value of the outer product right there in-place from the reshaped 1D array and the two indices. Keep it simple!

---

Here's how that could look like...

[altenbach]

---

@TgTgTg wrote:

Here is the algorithm, if you have matlab you can try it with NN=4, e and f values in {1,5,9,13}:

---

Sorry, I no longer read text code, but if that's all, the LabVIEW code should be significantly simpler. 😉

Do you have a website containing the relevant theory and formulas directly? Does the algorithm have a name?