Insert 2d array into 3d array

[Daikataro]

Greetings everyone, for datalogging purposes, I need to query data from a DAQ, add a timestamp and post-process data, then save it all to a .tdms file. I also need to create a data buffer for a certain event, which I intend to store using a 3D array, I only need 1 cycle of data, so it would have 2 pages, 1 current and 1 backlog. Problem is, for some reason, it is only storing a single row of data, instead of whatever amount I send from the post-processing loop, I can have a 2d array of 2000 rows per 6 columns, and it will only store 1 row, what am I doing wrong?

[altenbach]

Here it would definitely pay to use a fixed size 3D array and use replace array subset to replace the oldest plane. Also ensure that the data is in memory order, so use the correct plane orientation.

What's the point of the "cicle" local variable?

[Daikataro]

The thing is, array size would most likely vary, so array size would most likely not be fixed. Basically I intend to ask for 2 samples out of a 2kHz 2kSamples buffer every milisecond, the 1 second timer was just so i could see what the code was doing. I would like to fill up page 0, then when cycle is different from its last iteration (new cycle) move that page to page 1, empty page 0 and start over indefinitely, but my problem remains, I can only get one of the values into the array, instead of the 2000x6 or whatever amount of data I have

Yeah the local variable is pointless here, this is just an extract of a much larger code so I just copied the consumer-producer datalog part, and the variable came along, in the main code it's part of the main loop, so how can I get the array to take the entirety of the data, instead of a single point?

[RavensFan]

I don't see where you have a problem with your code.

Array is a 2000 x 6 array.  Array 2 is a 6 x 2000 array.

Circular Array is a 2000 x N x 6 array where is is continually growing depending on how many times the loop has run.

So I don't see where you are getting "only one of the values into the array".

[Daikataro]

Yeah I think I did not explain myself properly, allow me to recapitulate:

-I would like to insert the "array" part, a 2d array of 6 columns N rows (variable depending on amount of samples I get from DAQ) into a 3d array in page 0, appending row and column to the end of the array, so page 0 and only page 0 will continue to grow with more samples taken

-When a cycles depending event occurs, I wish to take the contents of Circular array page 0 (all other pages I want to be empty) and insert that into page 1, then clear page 0 into an empty array, and start the datalogging process again, with page 0 growing with the samples, and page 1 containing the past information as a buffer

The size of buffer is fixed in terms of pages (only 2, 1 for main log 1 for back) but I can't make the 2d array fixed, since amount of samples depends on user inputs and physical variables, so predicting size is impossible, my problem is that, if I leave insert into array unwired, it logs each iteration to a different page, if I wire row index, It logs one row of data in each page, and if I wire column, well data is unreadable. What would be the correct way to do what I need? That is, insert the 2d array into page 0 every time, then periodically dump its contents into page 1 to use as backlog.

[altenbach]

---

@Daikataro wrote:

The thing is, array size would most likely vary, so array size would most likely not be fixed.

---

I assume that the array size is fixed for each run, because you cannot have "ragged arrays". You you simply initialize the 2D array at the beginning and whenever the size changes. Alternatively, you can make the array comfortably large, don't use all elements in the plane, and keep size information elsewhere.  Your constant resizing is not efficient. All the array manipulations should operate in place.

[Daikataro]

Thanks for the heads up, but sadly, it is not fixed on each run, not even between iterations. See, I am sampling (currently) 6 sensors with a DAQ at 2kHz and 2kSamples per second. When the mechanical system is moving, I wish to monitor data and store exactly (can be a bit more info) one cycle as buffer in case an event occurs. The mechanical system takes a very variable amount of time to finish one cycle, because the user can manipulate movement speed and certain delays in real time, so a cycle can take anything from 5 to 40 seconds to finish, not to mention the obvious inconsistency in mechanical movement that can alter cycle time by about 400 miliseconds, give or take. So I would have a cycle of say, 12 seconds worth of information (24kSamples) followed by another of 11 (22kS), that's why I would rather have a dynamically populated array that is reset to a blank page after each iteration cycle. For convenience purposes, I wish to store all data corresponding to a single cycle in a single page, so if the event occurs, I can just dump all the info inside the buffer to a .tdms and notify the user.

If I were to use a fixed array size, what would I have to do? How do I populate it with the constant new info? Would I get the behavior I want, that is, all data of a cycle into a single page?

[altenbach]

If the sizes are not fixed, use a 1D array of clusters, where each cluster can contain a 2D array of different size.

If the sizes are fixed, here's a quick draft to show how to replace the oldest plane with the newest data, all operating in-place. You can easily calcualte the current indices (t, t-1, t-2) from [i]. Note that the array elements for each plane are adjacent in memory.

[altenbach · Accepted solution]

---

@altenbach wrote:

If the sizes are not fixed, use a 1D array of clusters, where each cluster can contain a 2D array of different size.

---

Here's how that could look like... (of course memory allocations are more inefficient here)

[Daikataro]

Your approach with clusters looks really promising, and it might be the functionality I need, but we're back to square one with the current approach, that is, the cluster of 2D arays is single-use, and what I need is being able to append new data to the previously stored one, then moving it to the buffer at new cycle. I modified the code to use the array of clusters of 2D arrays (wow, that's a mouthful) but the data manipulation feels a bit Goldbergish, I don't know if there's an easier or more ellegant way, but basically what I do is:

-Index element zero from array (my main storage)

-Unbundle said element, I get a 2D array of string

-Insert processed data into array

-Re-bundle array so it becomes a cluster again

---Up to this point it works fine, I see the data I want but then---

-Use replace array subset to replace the cluster containing the old information, with the updated cluster containing new data, it does nothing, gives me a completely empty array

If we could get this to work, I think that's the functionality I need for a buffer to work properly! What am I missing? Here's updated VI