Add Support for Array of Array

[SteveChandler]

In LabVIEW it is not possible to have an array of arrays. Something that gets you close is a 2D array but each row must be the same size. You can have an array of clusters of arrays to get around this limitation. Many languages do support arrays of arrays.

[Brian_Powell]

Back when we were rewriting everything for LabVIEW 2.0, we put some thought into this, but chose not to support it.  (Don't ask me why; that was 24 years ago. 😉

What are some of the use cases that aren't satisfied by the array of cluster of array?  In terms of memory storage, the array of cluster of array is exactly what you're asking for, so I think it boils down to having to do the cluster bundle/unbundle to get to the element.  Is there more to it than that?

This also opens up some ambiguities in the language.  Should output indexing tunnels on loops have an option to create an N-dimensional array of arrays, as well as an N+1 dimensional array of elements?  Are there parts of the language that get easier/more intuitive with an array of arrays?

[SteveChandler]

The only advantage is probably that I wouldn't have to use an array of cluster of array. Maybe if it is implemented this is how it would work internally. The cluster would be hidden and the bundle/unbundle would be transparent.

Those are some interesting thoughts about output tunnels. It is something I will have to think about. I hope that others can comment on that.

[yenknip]

From reading your spec, Steve, it suggests to me that as each element in the top level array is unique, I would ultimately create an array of classes, or some other wrapper for the arrays - like you are forced to do with clusters,

I do understand where you're coming from; but for an array to wrap around a number of elements, the elements must have the same datatype. If each element is an array with different numbers of elements, then the elements are not of the same datatype.

I'll have to think about this a bit before I kudo it or not

[AristosQueue (NI)]

I asked about this when I started at NI... which was almost exactly half the time since LV 2.0, so the memories are only half as dim. 😉 Here's what I was told -- and it seemed reasonable: First and easiest to possibly address was differentiating these two different kinds of 2D array visually on the diagram. More thornier issues lay with implementation: certain primitives become more complicated -- Build Array would need pop up options to decide if the two 1D arrays should become a regular 2D array or a ragged 2D array. Or we would have a second Build Ragged Array primitive. Either solution just increases the complexity of the array API, which is really nice in its elegant simplicity. Is it really an advantage to have to explain the difference between them when the array-of-cluster-of-array solution is performance-wise identical to the array-of-different-sized-arrays solution? There just didn't seem to be a compelling reason to add another concept to the language.

I never dug into it much after that early conversation. It is very rare that I have needed/wanted ragged arrays, and in those cases, the array-of-cluster-of-array has always been both sufficient and easy enough to implement, for me at least.

[SteenSchmidt]

I concur with Stephen here; I've needed array-of-arrays on only a few occasions in the past 12 years or so I've worked with LabVIEW, and the horrible mangling of the array API needed to support this would be a pain to work with in all the other cases when I use arrays (and that's a bit more than "a few" times obviously ).

The array-of-clusters-of-array has the positive sideeffect of real simple upgrade of the data type - typedef the cluster and you can easily drop in additional elements if your inner data needs company.

I'm a bit more sceptical on the array-of-arrays being identical internally to array-of-clusters-of-array - is that true memory-wise in LabVIEW? I've worked for 11-12 years with the embedded code for HP graphical/symbolic calculators (NEC Saturn and ARM9 processors). Data needs a prologue/header to tell the type, then for dynamic length data a length field, and then the data. The length field of a composite object like an array would be the number of elements, and the data would then be the elements one after another again each with its own prologue, length and data. This considering contiguous objects which would be true for almost all machine representations of native types (linked lists or sparse matrices aren't considered native types).

Example: memory representation of an array-of-arrays-of-DBL;

Array-of-DBL, the inner type:

<array_prologue><num_elements><dbl_prologue><dbl_data>...<dbl_prologue><dbl_data>

Alternative representation with advanced type support, a special type specifier for composite objects of constant size data elements (uses up less space since you can omit the prologue for each element when the element size is constant):

<fixed-element-size_array_prologue><num_elements><element_prologue><dbl_data>...<dbl_data>

If we stay with the first case the outer array would look like this:

<array_prologue><num_elements><array_prologue><num_elements><array_data>...<array_prologue><num_elements><array_data>

But if you represent that with an array-of-clusters-of-array-of-DBL, you'd get:

<array_prologue><num_elements><cluster_prologue><num_elements><array_prologue><num_elements><array_data>...<cluster_prologue><num_elements><array_prologue><num_elements><array_data>

That is, one <cluster_prologue><num_elements> more per outer array element than the array-of-arrays representation. The only way to get around this is to have a special prologue like <array-of-clusters_prologue>, thus enabling you to skip the cluster prologue for each inner element, but that would mean a huge (probably unexpected) increase in data size of you added even a Boolean to your cluster, because then you would have to revert from the <array-of-clusters_prologue> "trick" to the full-blown <cluster_prologue><num_elements> per outer array element.

How does that stack up with LabVIEW reality?

Cheers,

Steen

[AristosQueue (NI)]

I think the simple answer is: There is no cluster prologue. The data size of a cluster is exactly the same as the sum of the data size of its elements. We don't need a number of elements in the data because that's part of the data type, known at compile time.

[James_W]

If this feature exists already, why is is so difficult to add support for array of array?

surely it just means handling array of cluster of array under the hood!!!

I've come accross this problem many times but never been thinking warped enough that I must use array of cluster of array instead of array of array, instead I end up having to handle the nasty default data value in a 2D array to know when the minor array within the major array is terminated and of course this can produce errors if not everything is handled perfectly.

Cluster to array means we must know our array size before we unbundle so you have to be thinking really warped to think of setting this to 1 and being able to use array of cluster of array.

This is simple for NI to fix under the hood, but not so obvious when you are programming at 7 at night.

James

[SteenSchmidt]

@Stephen: Ah, I see. The cluster is only used at edit-time to compound elements, so the IDE displays them more or less as an aid for the programmer - 'clusters' never make it to the object in memory itself. The compiler simply, as clusters are just contiguous chunks of memory, point to the inner elements themselves by the starting handle/pointer + an offset to hit the inner element. Clever, and obvious at the same time . In that case array-of-cluster-of-array is exactly the same object as array-of-array would be.

@James W: It's not difficult at all to add support for array-of-array, as Brian tells us in the first comment. But as Stephen correctly points out, it would make a mess of the array API. Think about the build array function for instance. Try dropping that into an otherwise empty block diagram, and then wire two 1D-arrays to it. How would you tell the build array function to in one case make a 2D-array and in another case make a 1D-array of 1D-arrays? Or would you prefer a completely separate array API only for array-of-arrays, effectively doubling the number of array functions?

The array-of-cluster-of-array method is more than adequate for me the few times I need array-of-arrays. Much more palatable than gettting served a mangled array API every single day.

Cheers,

Steen

[Brian_Powell]

When I was younger, I thought a lot about the design of programming languages.  I also did a lot of studying of algebraic structures, which have many parallels to programming language design.

In my opinion, an "array of arrays" ought to be interchangeable with an "array of one dimension larger".  If you don't do this, the language (the constructs used to manipulate an array) get messy.  They are messy to implement under the hood, and messy to have the user describe their intent.  You end up with an inelegant algebra.

Pointer-based languages like C do have two different representations... multi-dimensional arrays (contiguous block of memory), and arrays of pointers.  My experience (tainted by our style of programming in LabVIEW) is that I approximately never use multi-dimensional arrays in C; I do my own pointer arithmetic on either a one-dimensional index or on a pointer to an array element.  C isn't known as a mathematically elegant language; it's known as a "you can do anything you want" language.

I'm not sold on this being a good idea, but maybe someone can show me examples of how this simplifies the language.

[SteveChandler]

Well I surely wouldn't want to mangle the nice API. I would of course only want this idea to be implemented if it does not require a mangled API.