Hello, I am working on a Machine Learning application where I need to work with very large arrays, primarily 1D String arrays, and as the system works, the arrays expand and then there are integer arrays which start from 1D but then become 2D (can have 3-4 or more columns) and around a million rows. Since these integer arrays are to store 0's and 1's, I use U8 integer so that it uses less memory. I am using LabVIEW 2013 SP1 (hoping to get my hands on LabVIEW 2014 in a few weeks), on a Windows 8.1 Pro, Intel core i7-3537U CPU@2.00 GHz, 2.50 GHz (64-bit OS and x-64 based Processor), 8 GB RAM. There are several issues: 1) Speed issue: I will explain this with an example. During the runtime, there is a 1D String array (let's call it "X_str") of around 550,000 elements which looks up each of those elements in another 1D String array (let's call it "T_str") of >3 Million elements, and when a string is found in T_str, the match index is looked up in another array 1D U8 array (let's call it "T_val"), and the corresponding value (either 0 or 1) is stored in the position of X_str into another 2D U8 array called X_val. So for example if a string "aaaa" is in X_str[450], it's looked up in T_str[] and is found at T_str[135000], then the value at T_val[135000] is fetched and is stored in X_val[450,column id] (either in column 1 or 2 ... as per the cycle). Now, I am doing this in the simple manner, without using the "subroutine" priority. What could be an effective way to do this? So that the speed is fast. The program in "Update Arrays.zip" shows my implementation of this part. 2) Memory issue: The program stopped at a very critical juncture, while almost producing the results, because it ran out of memory. It ran out of memory for copying the array as well as writing it to a file. The method I am using to write to file is in the attachment (Write to file.zip). The application has given some positive results when the examples were not very complex in terms of size. But a slightly larger example strings created huge arrays that I cannot even copy them as constants into another file for analysis. The memory is full. I am trying to redesign the algorithm so that many unnecessary steps are avoided, but I also want to know what can be improved in terms of my LabVIEW usage. I am also exploring how Data Value References and In Place Element Structure can be useful in this case. Thanks a lot for your time!

LabVIEW

Memory Management for Large Arrays

Author

@altenbach wrote:

Here is a quick attempt at a variant version. The order ouf the output is different, but that might not matter. The entries are still correctly matched to the values. If it matters, a few things need to change.

Of course you would carry the data inside the variant across caller and subVIs, etc. There is not need to extract the actual data until you save.

Ok, so while implementing this method in one of the functions, I came across a situation when I have to compare the values (rows) of one 2D array with those of another 2D array.

I am converting all the combinations of String arrays and corresponding U8 2D arrays into variants. But since I cannot search "values" of attributes in a variant, I have to fetch their String and Values array sets (Variant to Data).

I have used the example you described, but the values are now a 2D array, unlike the 1D T(val) out in that example. And the problem arises in fetching those row values from the variant. The Variant gives output values as 1D variant array, but my desired output of the values is a 2D numeric array.

I could not fetch the 2D Array data from the variant attribute values. Instead I tried for 1D array and it worked (just to see how the order of the Str elements changed).

But in order to perform the operation I need (two U8 2D arrays, and searching if rows of one of the arrays are there in the other one), I cannot work with variant because variant gives data by the name of their attributes only, not the attribute by the values (of course, that doesn't make sense, but even if I can find the first non-matching row and its corresponding attribute name, that's enough).

The row index output in the above program is used to fetch the corresponding string from a string array (which I am planning to use as variant attribute).

For all the other operations, Variant option is very good, performance wise. So I have decided to go for it. But this particular functionality has encountered a road block, and I might have to stick to the arrays if I cannot "either fetch back the 2D array from Variant values, or find a way to compare attribute values of two variants".

Thanks ahead!

Vaibhav