LabVIEW
Store the Data
You have two main problems:
(1) The overuse of global variables.
(2) Constant resizing (growing) of arrays.
If you know the final 2D array size, initialize an array with that dimension, then replace rows as you go. At the same time, you should also get rid of all global and local variables, they are not needed and will multiply the use of memory and cpu resource, slowing you down. Global variables should be used sparingly and mostly for UI issues. In a well designed proram, they are rarely needed. Large data arrays don't belong in local or global variables.
Attached is a simple example, and it only takes a few seconds (mostly to calculate the sine function 1000000x in this case).
(1) The overuse of global variables.
(2) Constant resizing (growing) of arrays.
If you know the final 2D array size, initialize an array with that dimension, then replace rows as you go. At the same time, you should also get rid of all global and local variables, they are not needed and will multiply the use of memory and cpu resource, slowing you down. Global variables should be used sparingly and mostly for UI issues. In a well designed proram, they are rarely needed. Large data arrays don't belong in local or global variables.
Attached is a simple example, and it only takes a few seconds (mostly to calculate the sine function 1000000x in this case).
I'm sure there is a better solution, even if the 2D array is being filled from various modules at various times.
A single copy of your array in memory is already about 80MB (if DBL) and if you are not careful with coding, you might easily generate 10 copies of it and constantly copy the entire thing back and forth between different locations.
Have a look a application note 168: LabVIEW Performance and Memory Management
. Every read operation from a global variable generates a copy of the data!
(Bart Simpson (as a LabVIEW programmer) would probably need to write the following quote 100x on the blackboard: "Do not overuse glo
bal and local variables when working with arrays or strings. Reading a global or local variable causes LabVIEW to generate a copy of the data.") 🙂
I would store the full array in a "functional global" (LV2 style global), i.e. in a shift register of a subVI. Keep the insert point in another uninitialized shift register. Notice that each "append" call only requires transfer of a single row, the calling VIs thus don't need to read out an extra copy of the full huge array for each append operation. Place a case structure inside the storage VI where you can select various modes (initialize, append, read out, save to disk, get average of column x, etc.) such that most operations remain local to the "functional global" and don't need shuffling of huge arrays back and forth.
How often do you need access to the full array? Another possibility might be to go directly to disk and append each row as a datalog record. It will be easy to later retrieve a few interesting row and memory us
e will be very light.
A single copy of your array in memory is already about 80MB (if DBL) and if you are not careful with coding, you might easily generate 10 copies of it and constantly copy the entire thing back and forth between different locations.
Have a look a application note 168: LabVIEW Performance and Memory Management
. Every read operation from a global variable generates a copy of the data!
(Bart Simpson (as a LabVIEW programmer) would probably need to write the following quote 100x on the blackboard: "Do not overuse glo
bal and local variables when working with arrays or strings. Reading a global or local variable causes LabVIEW to generate a copy of the data.") 🙂
I would store the full array in a "functional global" (LV2 style global), i.e. in a shift register of a subVI. Keep the insert point in another uninitialized shift register. Notice that each "append" call only requires transfer of a single row, the calling VIs thus don't need to read out an extra copy of the full huge array for each append operation. Place a case structure inside the storage VI where you can select various modes (initialize, append, read out, save to disk, get average of column x, etc.) such that most operations remain local to the "functional global" and don't need shuffling of huge arrays back and forth.
How often do you need access to the full array? Another possibility might be to go directly to disk and append each row as a datalog record. It will be easy to later retrieve a few interesting row and memory us
e will be very light.
Try using a preallocated 2D array of fixed size inside the LV2 style global and use replace array subset instead of build array. (Similar to my demo above).
