Sort 2D Array and Convert into 3D Array

[JoeWork]

I am having an issue creating a VI to convert a 2D array into a 3D array based on a value from a column in the 2D array.  I read a CSV spreadsheet in and determine how many distinct values are in the column of interest.  I strip 1D arrays off of the 2D array and then use the value from a column to determine what array page in a 3D array that 1D array needs to go into.

I am trying to find a way to build this 3D array dynamically so that it is done quickly and doesn't use up memory.  I have included it in this post.  If you have time to review it and provide some guidance I would appreciate it.

Thanks.

[syrpimp]

wr

[johnsold]

I suspect that you could do this with less code.

Can you post a small data file along with the expected result?

General comments:

1. Use Array Size, not Matrix Size. In LabVIEW the matrix datatype shold only be used for specific matrix operations which cannot easily be done by array functions. All you have is simple arrays of strings.

2. Array Subset is generally a better choice than Delete from Array.

3. The upper for loop can be replaced by Index Array. When you wire a multi-dimensional array to Index Array it expands to give you an index input for each input. Just wire the 2 to the column index input.

4. You already extract the second column in the upper forl loop and then you do it again in the lower loop. Just pass the data. Also Decimal String to Number is polymorphic. It will accept an array of strings and produce an array of numbers. 

5. Insert into Array is generally not a good choice. It may be required here, but I am not sure exactly what you are trying to do so it is hard to tell if there are better options.

Lynn

[JoeWork]

ere is the data file I was working with.

1. Use Array Size, not Matrix Size. In LabVIEW the matrix datatype shold only be used for specific matrix operations which cannot easily be done by array functions. All you have is simple arrays of strings.

       This is used because array is 2D.  I want size of rows and columns.

2. Array Subset is generally a better choice than Delete from Array.

       I am getting rid of header information in the original spreadsheet.

3. The upper for loop can be replaced by Index Array. When you wire a multi-dimensional array to Index Array it expands to give you an index input for each input. Just wire the 2 to the column index input.

      Done.

4. You already extract the second column in the upper forl loop and then you do it again in the lower loop. Just pass the data. Also Decimal String to Number is polymorphic. It will accept an array of strings and produce an array of numbers. 

      The input into the 2nd column extraction in the for loop is a 1D array.  I am trying to pull the single value out of it.

5. Insert into Array is generally not a good choice. It may be required here, but I am not sure exactly what you are trying to do so it is hard to tell if there are better options.

With this VI I want to read a CSV file and sort/seperate it into 2D arrays depending on the value in column 2.  So if there end up being 4 distinct values in column 2 for the whole CSV file, I should end up with 4 2D arrays that have a single value in its column 2.

I hope this better explains what I was trying to accomplish.  Basically my problem is I am having a hard time figuring out how to insert 1D arrays into pages of a 3D array.  A problem with that was figuring out how to do it dynamically so I don't use up a lot of memory.  Already had to shutdown down my computer due to a run away "Out of Memory" train.

Thanks for your help.

[johnsold]

I have not looked at the data yet.

1. Array size will work with arrays of any number of dimensions. For 2D and higher arrays the output is an array of sizes. It produces the same results.

2. Array Subset does that also.  Delete from Array is more useful when you will be using both parts.

4. Autoindexing on the numeric array should produce the same single values. I will let you know after I have looked at the data.

Lynn

[altenbach]

---

@JoeWork wrote:

I am trying to find a way to build this 3D array dynamically so that it is done quickly and doesn't use up memory.  I have included it in this post.  If you have time to review it and provide some guidance I would appreciate it.

---

Please include some sample data file (or run your VI until "all rows" contains data, then change that control to a constant (right-click...change to constant), delete the file IO, and attach the VI once more).

Also include the subVIs, not everybody has openg tools installed.

I also don't understand why you use a %d format if you read the fields as strings. Makes no sense.

[johnsold]

The VI you posted moves only about 2% of the input data to the output arrays. (1000 rows X 4 pages from 209434 rows).  Why do you choose 1000 rows for the size of the output array? The values by which you select pages are not uniformly distributed. So it looks like you end up with the last 1000 data rows for each page but they may come from different parts of the input data.

[Edit: I put a serial number = row number of original array into the sixth column of the original data. The first and last serial numbers on each page are:

Page    First         Last

0         209432   203999

1         209430   204736

2         209433   207392

3         209419   203471           end edit]

Is this what you want?

Lynn

[johnsold]

Here is a version which produces the same results as yours (I think). I looked at the first and last 20 rows on each page and they match.  Your VI takes about 200 seconds on my computer. The VI attached takes about 10 seconds. If I set the Insert Serial Number to True, it adds about 5 seconds.

The changes:

- Use Array Size in place of Matrix Size.

- "Reverse" the array so that it is last row first. Since you only use a small amount of the data at the end of the original array, I get that first and then stop.

- Use Index Array and Decimal String to Number to get array of values from column 2.

- Use a while loop rather than a for loop for constructing the output array. It stops as soon as the output array is "full."

- Use an array of row indexes for each page in the output array. The value from column 2 indexes that array and indexes the Replace Array Subset to update the index used. In Range & Coerce is used to keep the values from exceeding the initialized size.

- Use both page and row indexes on the Replace Array Subset for the output array. This eliminates the Insert into Array which was in the original.  Add 2 to the row index to keep the first two lines blank. An alternative would be to prepend two blank rows to the output array after the loop finishes.

The Stop button and the Insert Serial Number button and code are to see which data was going where and for troubleshooting purposes. They can be removed.

Lynn

[johnsold]

While in bed last night (when I should have been sleeping), it ccurred to me that it is unecessary to "reverse" the 2D array first. This version indexes in reverse by subtracting "i" from the number of rows. It runs in less than 2 seconds.

The version posted hs a small bug: Page 0 has three blank rows at the top and it misses one element at the end. I suspect that an increment or decrement is needed on the index calculation but have not pursued it.

Lynn

[nyc_(is_out_of_here)]

---

@johnsold wrote:

While in bed last night (when I should have been sleeping), it ccurred to me that it is unecessary to "reverse" the 2D array first. This version indexes in reverse by subtracting "i" from the number of rows. It runs in less than 2 seconds.

The version posted hs a small bug: Page 0 has three blank rows at the top and it misses one element at the end. I suspect that an increment or decrement is needed on the index calculation but have not pursued it.

Lynn

---

Can you save the VI into LabVIEW 2013?

Thanks.