NI Home > Community > NI Discussion Forums

LabVIEW

Reply
Member
z crittendon
Posts: 15

what is the maximum size of an array?

Hi,

For the program I am writing, I need to be able to aggregate a certain amount of data before writing it to disk (due to the legacy format I am writing it in). I am planning to use an array to do this, but I need to know how much the array can hold before it is full.

On paper, the array seems to have a limit of 2,147,483,647 items (derived from the long-int fed into the 'dimension size' parameter of the "initialize array" function). However, when I tried to create and initialize an array of this size (using "initialize array", nothing happened. There were no errors, and the array was not created. When I tried creating a smaller (but still very large) array, I got the failure "fpsane.cpp," line 269.

So: Is there a
maximum array size? If not, is there a way I can tell at runtime what the largest array size possible is?

Thanks,
Zach Crittendon
Active Participant
JeanPierre
Posts: 1,056

Re: what is the maximum size of an array?

You will likely be limited by the largest block of free memory currently
available on your system rather than by LabVIEW. You will also need to be
very careful to prevent LabVIEW to make unncessary copies of the array.

What does actually prevent you to write data on disk as it goes? Or to
aggregate an amount of data in a file instead of in an array? Files are
random access too e.g you can access any location of a file as you can
index an array (well at least 2GB...). For large amount of memory, the OS
will manage this with huge disk swapping anyway.

Jean-Pierre Drolet


LabVIEW, C'est LabVIEW

Active Participant
Doug (CapeFl)
Posts: 285
0 Kudos

Re: what is the maximum size of an array?

General question: To minimize making extra copies of an array (if an application/VI needed such), would it be better to receive/send the array data via Reference (address/pointer) versus an actual copy of the array variable?
I believe wiring in/out an array as well as using a reference for array values is better than making local variable copies.
Doug
Active Participant
CraigGraham
Posts: 385
0 Kudos

Re: what is the maximum size of an array?

As far as possible, this is already done transparently when you pass an
array to and from a sub-VI.

I'm not sure what happens with locals if the panel is open. If the panel is
closed, then I'd assume pass-by-reference is also taking place. The reason
for my uncertainty is that panel items maintain their own copies of the data
if the panel is open. But certainly wiring direct is always faster than
using a local..

Doug (CapeFl) wrote in message
news:50650000000500000008440000-1000509150000@exchange.ni.com...
> General question: To minimize making extra copies of an array (if an
> application/VI needed such), would it be better to receive/send the
> array data via Reference (address/pointer) versus an actual copy of
> the array vari
able?
> I believe wiring in/out an array as well as using a reference for
> array values is better than making local variable copies.
> Doug
Active Participant
JeanPierre
Posts: 1,056
0 Kudos

Re: what is the maximum size of an array?

When using a reference to modify an array, you have to read the "Value"
property, an operation which makes a copy of the array the same way as for
reading a local or a global. You can't modify a single array element using
an array reference without reading the whole array. In LabVIEW, references
are references to LabVIEW objects (controls,indicators), not data pointers.
They are mainly intended to manipulate cosmetic/appearence properties of the
object rather than its data. LabVIEW is optimized for dataflow to pass data
from place to place in your diagram. As Craig pointed it out, passing an
array to a subVI is transparently done with a data pointer, provided that
this data does not need to be displayed e.g. its front panel is closed.
LabVI
EW makes data copies for display purposes.


Jean-Pierre Drolet


"Doug (CapeFl)" a ecrit dans le message news:
50650000000500000008440000-1000509150000@exchange.ni.com...
> General question: To minimize making extra copies of an array (if an
> application/VI needed such), would it be better to receive/send the
> array data via Reference (address/pointer) versus an actual copy of
> the array variable?
> I believe wiring in/out an array as well as using a reference for
> array values is better than making local variable copies.
> Doug


LabVIEW, C'est LabVIEW

Active Participant
Doug (CapeFl)
Posts: 285
0 Kudos

Re: what is the maximum size of an array?

Craig & Jean-Pierre: thanks for the info, Doug
Member
z crittendon
Posts: 15
0 Kudos

Re: what is the maximum size of an array?

I wanted to avoid writing the data to the disk in real-time for performance issues. I found that when I was writing data it would make the waveform display very choppy, and put me further behind in the buffer. So, I was hoping to just keep the info in memory until the run was finished (around 5 minutes/6 megabytes).
Active Participant
nestor
Posts: 668
0 Kudos

Re: what is the maximum size of an array?

Zach,

I will add my answer to the ones already posted, the array size is limited by your computer's memory... although I will tell you that this kind of huge size arrays are not a good programming technique, because even though your computer or LabVIEW may be able to handle such array, your performance and system memory will be affected. I've been looking at some application notes and your array size is "acceptable" for how LabVIEW manages to know the size of an array, but I will really recommend you to decide a solution different than that.

The answers posted already are very good, as an added suggestion think about using a database and accessing it from LabVIEW, instead of handling a so large array.

Good luck and thanks for the info from the other person
s very precise answers...

Nestor Sanchez
Applications Engineer
National Instruments
Nestor
National Instruments
Active Participant
Greg_McKaskle2
Posts: 1,994

Re: what is the maximum size of an array?

....
> So: Is there a maximum array size? If not, is there a way I can tell
> at runtime what the largest array size possible is?
>

As was already posted, your array will most likely be limited
by the amount of memory that your computer can address. The
next generation of OSes will have 64 bit addressing, but for
the moment a pointer in a process is only 32 bits, which means
that it is quite difficult to have an array with more than 2^32
elements in memory even if they are only a byte apiece.

If your array elements are 256 bytes, then you can expect the
OS to start throwing up major difficulties at around 2^24.

As you noted earlier, the size element that is a part of each
array and string in LV is a signed int32 and will impose one
limit on an array size. In addition, some of the nodes in
LV, like the array initialize node will not operate if wired
a negative number, and it should stop execution with an out
of memory dialog when run with really large numbers. In these
situations, LV will ask the OS for the appropriate amount of
memory, but if the OS doesn't have RAM or disk swap space, it
will fail to allocate the memory for LV, so LV will pass on the
bad news. It shouldn't lead to a crash or anything too bad, but
it will stop the program as many of the fundamental nodes have
no way to return this error to the diagram.

It sounds like for your application, you want to keep memory in
memory while being acquired. You don't state how much data that
is, but this is quite common. Preallocate an array in a LV two
style global. In your acquisition loop you can call the LV2
style global to store the elements one at a time, or more likely
a buffer at a time. In the LV2 style global, use replace subset,
or if not using LV6, loop and replace element and write the array
back into the shift register. After your data acquisition is
complete, call the global with a command to write the global out
to disk, and it can loop through producing the file or files for
you.

Like I said, this is not uncommon, and it will not make lots of
copies of your data since only the LV2 style global subVI even
has all of the data. The buffer will be preallocated to avoid
failing partway through and to avoid introducing timing delays
as the OS memory manager looks to move the block around as it
is resized. Your major concerns should be the amount to
preallocate, which is determined roughly by how long the
experement will run.

If for some reason you do not have an upper limit on the
length, another approach would be to make more than one
array stored by the global. Init the global to have and array
or cluster of arrays. The initial size is one array of cluster
of 10M lets say. Once the 10M is used up, make another element
in the array rather than growing the first one. It may not sound
that much different, but it is quite different to the OS as LV
keeps the arrays in contiguous memory.

Greg McKaskle
Member
z crittendon
Posts: 15
0 Kudos

Re: what is the maximum size of an array?

Thank you. I was already looking into using a global array, but was having difficulty figuring out how globals work in L.V.

One question I have, though, is what is a "LV2 style global?" How are they different from the globals found in the "structures" menu in LV6? And how do I create a LV2 style global?

Thanks,
Zach