Array Subset

[Strokes]

So this is not really a labVIEW specific problem, more a issue with my mind not working like a true programmer to find a solution to a (probably straight forward problem). 

I want to process some data (array of numerics). The array of data is overwritten every time new data is available, I've simulated this somewhat in the attached VI. Now here's the tricky part. The processing window is selected by the user. So if the length of the array(s) is 5 elements, and the user selects a processing window of 7, the array from the first iteration will have to be stored (this isn't hard using a shift register..) and the first 2 elements of the array on the second iteration will be appended to the array from the first iteration, then the processing will be done on the data.

I hope this makes sense, I've attached a VI of an awful effort of a solution on my part, but I've tried a few different ways about it and am getting nowhere close. I'm sure shift registers and/or feedback nodes, array size and case structures should be the main elements.

Let me know if more info is needed. 

[nathand]

In your VI, you are trying to combine arrays of different lengths into a 2D array.  This means in your case that the empty arrays are being expanded to 5 elements of 0, not sure if that's what you intended or not.

I'm confused about your idea of iterations and the data being overwritten each time new data is available.  How many points of new data are available each time?  Is the processing window always shorter than the number of new data points, or you may need to keep some old data in case the processing window is longer than than that?

It sounds like what you want is a circular buffer.  There are a couple of ways to implement one.  If you're just dealing with a 1D array, the easiest approach is to maintain a longer 1D array in a shift register.  When new data arrives, you replace the oldest data in the shift register array with the new data.  You can use either Rotate 1D Array (easy but less memory efficient) or an additional shift register storing the index of the next array index to replace (less memory and faster, but more code).

If you can provide a better example of your goal - either a VI showing the desired outputs as well, or just some simulated data typed into a forum reply - it will be easier to help you.  Right now I can't tell what sort of processing you're doing and whether your desired output from the VI is a subset of the array, several array subsets, or a single value that's the result of processing.

[Strokes]

---

@nathand wrote:

In your VI, you are trying to combine arrays of different lengths into a 2D array.  This means in your case that the empty arrays are being expanded to 5 elements of 0, not sure if that's what you intended or not.

I'm confused about your idea of iterations and the data being overwritten each time new data is available.  How many points of new data are available each time?  Is the processing window always shorter than the number of new data points, or you may need to keep some old data in case the processing window is longer than than that?

It sounds like what you want is a circular buffer.  There are a couple of ways to implement one.  If you're just dealing with a 1D array, the easiest approach is to maintain a longer 1D array in a shift register.  When new data arrives, you replace the oldest data in the shift register array with the new data.  You can use either Rotate 1D Array (easy but less memory efficient) or an additional shift register storing the index of the next array index to replace (less memory and faster, but more code).

If you can provide a better example of your goal - either a VI showing the desired outputs as well, or just some simulated data typed into a forum reply - it will be easier to help you.  Right now I can't tell what sort of processing you're doing and whether your desired output from the VI is a subset of the array, several array subsets, or a single value that's the result of processing.

---

Thanks for taking the time to attempt to understand my problem. Maybe I can describe the problem better.

In a producer loop, I'm producing an array of numeric values (this array's size is not fixed but it's always 1D). On every iteration of the producer loop the elements of the array are overwritten. Example outputs of the producer loops array is below.

Iteration 0: [1,2,3,4,5]

Iteration 1: [6,7,8,2,7]

Iteration 2: {2,4,6,1,1]

In a consumer loop a user specified numeric value which can be changed programatically through the use of a control decides the length of the window with which to process the data acquired. So for instance if the window is 4 elements, [1,2,3,4] will be processed to output an array of the same length while element four with a value of 5 will used along with the first three elements of the second iteration of the producer loop array [6,7,8] for the same processing and so on and so on.

Similarly the window could be 7. So in this case no processing would be done until a 7 elements of data were collected. So in my example the array in Iteration 0 would be stored in a shift register and the first two elements of Iteration 1 would be appended to it and then the 7 elements would be processed. Then another 7 elements would have to be collected before processing would be done again...

The processing could be anything like differentiating/integrating etc. and is not really important for the problem.

I'm sure there's a solution to this problem so hopefully I've painted a clearer picture.

[stevem181]

Hi,

See if the attached example does what you need.

One thing not addressed in the example is the potential impact of the Build Array function.

If the array is allowed to grow unchecked, eventually more memory will have to be allocated to store it.

This will impact performance.

steve

[nathand · Accepted solution]

In the consumer loop, start with a shift register containing an array.  Every time new data arrives append it to the beginning of the array.  Then iterate through subsets of the array until the array length is less than the processing size.  See attached image (similar to the approach posted by stevem181, his reply appeared while I was working on this).

[Strokes]

---

@stevem181 wrote:

Hi,

See if the attached example does what you need.

One thing not addressed in the example is the potential impact of the Build Array function.

If the array is allowed to grow unchecked, eventually more memory will have to be allocated to store it.

This will impact performance.

steve

---

Cheers Steve, it's definitely  a solution to the problem provided. However, due to possible high data acquisition rates, the array stored in the shift register might get very big very quick and reduced performance significantly which you have suggested. I would prefer to clear the data once it's been processed rather than hold onto it.

For that reason, I think nathand's solution is more suitable.

I appreciate the help though

[Strokes]

---

@nathand wrote:

In the consumer loop, start with a shift register containing an array.  Every time new data arrives append it to the beginning of the array.  Then iterate through subsets of the array until the array length is less than the processing size.  See attached image (similar to the approach posted by stevem181, his reply appeared while I was working on this).

---

Good stuff Nathand, seems to be just what I'm looking for. Top man

Strokes

[Strokes]

---

@nathand wrote:

In the consumer loop, start with a shift register containing an array.  Every time new data arrives append it to the beginning of the array.  Then iterate through subsets of the array until the array length is less than the processing size.  See attached image (similar to the approach posted by stevem181, his reply appeared while I was working on this).

---

Hi Nathand,

I rejigged your vi and it works as required if the period/window size is equal to or greater than the size of the data array. However, if the period/window size is less than the array it has trouble.

Example:Producer output [1,2,3]  Period size = 2. The VI performs the processing (just a multiplication operation in the attached VI) on [1,2] and forgets about the [3] and moves on to the next producer output. This leads to a significant loss in data.

Any ideas? 

[nathand]

---

Strokes wrote:

Example:Producer output [1,2,3]  Period size = 2. The VI performs the processing (just a multiplication operation in the attached VI) on [1,2] and forgets about the [3] and moves on to the next producer output. This leads to a significant loss in data.

---

No offense, but it's easy to prove that this statement is incorrect: put a probe (or an indicator) on the wire going to the shift register on the right side of the for loop.  Slow down the loop, or run with execution highlighting, and you'll see that the unprocessed elements remain in the array exactly as expected.  Perhaps there's some other misunderstanding about how this works?  Note that this will compute some intermediate results so quickly that you won't see them - consider putting a wait inside the inner while loop if that's an issue.  Let say the input is sequential numbers - the first input will be [1,2,3], the next [4,5,6].  The first calculation will operate on [1,2] leaving [3].  Then on the next for loop iteration it will operated on [3,4,5,6]; it will calculate the result for [3,4], notice that there are still enough remaining elements to do the computation again, and process [5,6].  This may happen fast enough that you won't see the result for [3,4].  If you were to do only one calculation per input and the input array is larger than the processing size, the array would grow indefinitely.

[Strokes]

---

@nathand wrote:

---

Strokes wrote:

Example:Producer output [1,2,3]  Period size = 2. The VI performs the processing (just a multiplication operation in the attached VI) on [1,2] and forgets about the [3] and moves on to the next producer output. This leads to a significant loss in data.

---

No offense, but it's easy to prove that this statement is incorrect: put a probe (or an indicator) on the wire going to the shift register on the right side of the for loop.  Slow down the loop, or run with execution highlighting, and you'll see that the unprocessed elements remain in the array exactly as expected.  Perhaps there's some other misunderstanding about how this works?  Note that this will compute some intermediate results so quickly that you won't see them - consider putting a wait inside the inner while loop if that's an issue.  Let say the input is sequential numbers - the first input will be [1,2,3], the next [4,5,6].  The first calculation will operate on [1,2] leaving [3].  Then on the next for loop iteration it will operated on [3,4,5,6]; it will calculate the result for [3,4], notice that there are still enough remaining elements to do the computation again, and process [5,6].  This may happen fast enough that you won't see the result for [3,4].  If you were to do only one calculation per input and the input array is larger than the processing size, the array would grow indefinitely.

---

My bad, I had a wait function inside the first loop and forgot about the inside loop. Appreciate the help.

Note to self: Use the highlight execution function more!

It really is a very neat solution to the problem suggested so thanks once again