Select indices/part of array with help of other array

[partial81]

Dear forum users,

Since I am very bad at programming, I can’t solve the following problem by myself:

I have two 1D-arrays, a main and a help array. In both arrays there are numbers (times in e.g. ms).

With the time information given in the help array, I have to read back the index of corresponding time values of the main array.

Since it is a bit complicated to understand what I want to have, I attach an Excel table as support.

As can be seen from the table, in column A the main and in column B the help array are given, and in column C a time interval.

Now I do the following:

I add the time interval (10 ms) to the times of the help array (not shown in the table). All times of the main array which are between these times, that is between 5 and 15, 30 and 40, 55 and 65, etc. are of interest for me. Since I do not exactly need the times of the main array, I just write down in column G in a new array the corresponding indices.

The array shown in column G is the object which I would need for further treatment in my main program.

Now I wrote a simple program in LabView 2011 which almost does the thing I want to have (see also attachment).

But there are 2 problems:

1: I get a matrix instead of a 1D-array (but with the right indices).

2: The program is extremely slowly! I fear it can’t be written worse as I have done it...

To see how bad my program works, I attach also an additional file: The main and help array as shown in the Excel table can be used for a quick test of my program (just put them into two files), a similar file with bigger arrays is attached. With this latter file it can be seen well how slowly my program is. All you have to do is to extract the 2 columns out of this file and to save them separately.

Who can help me to solve these two problems?

I think problem 1 can be solved easily with the right knowledge.

But problem 2 is really crucial in my opinion. Some of my arrays have sometimes several millions of lines! I fear some advanced programming skills are needed to write a fast working program. So the program should not work all the time with the whole long main array:

E.g. after the program has found the indices 7, 8, 9, it should stop at index 10 (the corresponding time value (40.0531) of main array is bigger than the upper limit of the time interval) and should continue to search for the next indices starting at index 10 and not 0.

Perhaps with some ideas like this a good program can be written. But unfortunately this is far above my programming skills at the moment.

I hope some people here feel the challenge and are willing to help!

I would appreciate every hint and help!

[AndrewAlford]

See attached for a possible solution, I have also attached a txt file with your data.

[MoReese]

Can you down-convert your VI to LV10 (Files>>Save For Previous Version)?  LV takes awhile to grasp, even if your are fairly familiar with text-based languages, it can be daunting at times because you have to learn the LabView way.  A novice programmer would have even more difficulty.  I would suggest reading the tutorials and help videos on NI's website for assisance.

[partial81]

Dear Andrew,

Thanks a lot for your answer! I am really happy that you read my question and tried to find a solution. Your program is really nice, but unfortunately it is almost as slowly as my own one.

First, I had to put the waiting time in the loops back to 0, otherwise the program takes “hours” even for small arrays.

Next, your program takes into account my first suggestion (it compares the main value to the allowed maximum help value. If the value is greater, it takes the next help value). That is really perfect! I didn’t know that it is possible to stop For-Loops in this way.

But then the program starts at index 0 of the main array all the time instead of starting at the index at which it stopped the inner For-Loop. That makes the program very slowly, especially for larger arrays.

Therefore, I tried with an additional variable to implement that the main array is not taken all the time from index 0 (see CorrelationExampleLV2011_2.vi in the zipped attachment).

That increased the speed of the program much! But it still does not work perfectly.

In the way I have tried to improve your program, some indices (34, 36, 39, 41, 44, 46) are not stored in the array. I know that this comes from my programming error, but I do not know how I can fix this error.

Therefore, I attach a new excel file in which I explain step for step how I think the program should work (take especially a look at line 48). Perhaps you and some others have time to look at it and some ideas how this algorithm can be implemented in my program. Perhaps there is also a faster way as I propose in my excel file.

I also attach the programs for LV2009 (should be easily convertible to LV2010 or LV2011) for Reese (Dear Rease, thanks for your supporting words. Unfortunately, I have to mention that I am also not familiar with text based languages).

I also add again two data files. A smaller one to test how the program should work (the content is the same as in the excel file), and a bigger one to test if the program is fast.

I hope these files and my little improvement of Andrews’s program can help to find a final solution.

[.aCe.]

Hello,

Not sure if this is what you are looking for, give it a go.

I have made a couple of changes, I have initialised the array and used replaced build array with replace array subset. I was told by an NI instructor that build array in a loop is a big NO NO.... hence the change.

The other change is on array subset, I have set a block size of 100 as working with large arrays slows the vi...

I have also accounted for the help array and the main array index change.

Hope it helps.

[MoReese]

---

@.:aCe:. wrote:

Hello,

 I was told by an NI instructor that build array in a loop is a big NO NO....

---

Well, you can use loops to init arrays as well as build arrays (auto-indexing).  What do you mean by "buiild array?"  Can you be specific?

[Ben]

---

@MoReese wrote:

---

@.:aCe:. wrote:

Hello,

 I was told by an NI instructor that build array in a loop is a big NO NO....

---

Well, you can use loops to init arrays as well as build arrays (auto-indexing).  What do you mean by "buiild array?"  Can you be specific?

---

Building an array in a while loop should be avoided since LV can not predict how large the array will be and when it grows beyond its current allocation, it has to allocate more memory resulting in porr performance.

Building an array on the edge of a foor loop is OK since LV can allocate the output buffer sized based on the For loop iteration count.

Prior to lv 6.1 we had to pre-allocate and replace in all loops but in LV 6.1 NI enhanced the allocation logic for the For loop.

Ben

[partial81]

Dear .:aCe:.,

Thanks a lot for this answer. This is a big forward step!!

I think it is a good idea to initialize the array and to use 'Replace Array Subset' instead of 'Build Array'. I also read that latter should be avoided in loops.

I see that the block size of 100 increases the speed of the program too. But that can also causes problems. I attach a new file (dataSmallNew.txt)  which does not work for this block size. Sure, I have to change the block size depending on my files. But isn’t there a smarter way? Ab initio I do not have an idea how big the block size has to be so that the program works (with the new file it starts to work with a block size of 251, but then still indices are missing), and at the moment I see no way how I can calculate the block size (there are dozens of possibilities how my files can be).

If I use the algorithm as I tried to explain in my excel table (ExplanationAlgorithm.xlsx), I also do not take a look at the whole array. In my mind I split the 2D array up in two 1D arrays. And then I check if the first value of the main array is in the first interval given by the help array, then I use the second value of the main array and so on. This is also mainly done in my CorrelationExampleLV2011_2.vi (except of a little error).

Interestingly, although I am using Build Array in my program and the whole array, it is much faster than yours for the big file (if I use large block sizes). Why? Is avoiding my little programming error (some indices are missing) so time consuming?

Well, I hope your nice program can be improved in such a way that it is faster and works without a manual tuning of the block size.

Anyway, thanks again! I am still hoping for some other nice feedbacks!

[partial81]

Dear forum users,

After working for some time, I think I was able to fix the error in my CorrelationExampleLV2011_2.vi program.

For the people which are interested in it, I attach my solution. There are two versions; the only difference between them is that one uses 'Replace Array Subset (RAS)', the other 'Build Array (BA)'. Both versions produce the array I want to have (in slightly different form), and both are fast (CorrelationExampleLV2011_2RAS.vi about 30 ms faster for the big file than CorrelationExampleLV2011_2BA.vi).

I hope some of you have fun to test the programs and to give some feedback.

If there is space for improvement, do not hesitate to mention it!

And if there is a more sophisticated solution, do not miss to post it!

I also want to emphasize that I am very happy about all replies and suggestions which have been given here so far!

[MoReese]

---

@Ben wrote:

---

 

Building an array in a while loop should be avoided since LV can not predict how large the array will be and when it grows beyond its current allocation, it has to allocate more memory resulting in porr performance.

Building an array on the edge of a foor loop is OK since LV can allocate the output buffer sized based on the For loop iteration count.

Prior to lv 6.1 we had to pre-allocate and replace in all loops but in LV 6.1 NI enhanced the allocation logic for the For loop.

Ben

---

Yes, I would agree concerning building the array with a While loop, but I was referring to For loops.  I should have explicitly mentioned that. I have never built an array using a while loop and see no need to do so.  But on that note, you could use the loop iteration terminal to control the size of your array.  Do you see any drawbacks to doing this?

Thanks Ben.