What is the most efficient and fastest way to deal with numerous large arrays

[jschmidtengler]

---

wrote:

Hi Julian,

would break down (to speed things up) the 4D array in two 3D arrays which I would pass to a Build Cluster Array Function

I don't think this goes the right way.

Instead it would help when you would provide a "real-world" example of your requirements (which kind of arrays, how many, …)!

---

Yes good idea. I prepared a sample VI that shows how I am doing it now. Suggestions on keeping it that easy to read and more RAM friendly are appreciated very much. 

Best,

Julian

[GerdW]

Hi Julian,

running your VI without those intermediate array indicators needs 300MB memory.

Remove unneeded indicators as much as possible as they create their own buffer…

bundled into an DataBundle Type and passed from vi to vi in measurement loop. In the loop the  bundle is unbundled when needed and operations on the arrays included are performed. Once finished the arrays are bundled together again and passed to the next vi.

When you "just" wire the data from one to the next subVI you could use the already mentioned DVRs.

Then you only need to keep a cluster of "DVR pointers" instead of the whole arrays. The subVIs still can access the data through those DVRs - it seems you don't use much parallelism in your code…

[altenbach · Accepted solution]

---

wrote:
I prepared a sample VI that shows how I am doing it now. Suggestions on keeping it that easy to read and more RAM friendly are appreciated very much. 

---

Yes, I would go with DVRs, so this is just a more general comment:

First rule, name your cluster elements, especially if it is a typedef. Now you can easily work on one cluster element without all that extra wiring of the untouched elements using an IPE. It is also much more self-documenting

(I still have the feeling that the same could be done with much less data structures.)

(Also, why are the arrays DBL instead of SGL? the 22bit mantissa should be plenty for 16 bit values. Half the memory!)

[jschmidtengler]

Thanks for the remarkable support! DVR's and IPE Structure did the trick for me! I would also like to thank you for the advice that goes beyond the question I had! 

Best,

Julian

[Ben]

---

@jschmidtengler wrote:

Thanks for the remarkable support! DVR's and IPE Structure did the trick for me! I would also like to thank you for the advice that goes beyond the question I had! 

Best,

Julian

---

I think it was Jim Kring that posted many years ago something to the effect of "a four dimensional array has no meaning." ever since then I have been reminded about what Jim wrote and found a better way of storing data.

Ben