Scanning large string arrays

[Eric1977]

Hello guys,

Could someone take a look at my code and suggest things that could make it faster? I am bringing in a rather large 2D array (~85000x71) and I am extracting two columns from the array & doing some comparisons against. There are 4 files total each broken out in 6 hour increments. One of the files is provided in CSV format. I feel like there is room for improvement. Could someone take a look and suggest alternatives that may be faster?

Thanks,

Eric

[altenbach]

I'd just autoindex over the full array and do all comparsons at once. You can wire strings directly to a case structure and make one case "0" and the other the default.

Instead of "insert into array" of two adjacent columns, you could just use "array subset instead.

[crossrulz · Accepted solution]

I would probably do something like this.  Since each loop is doing the exact same thing, you might as well just make it a subVI so that you only have to edit it in one place.  Be sure to set the VI Properties to execute as a reentrant VI.

[altenbach]

Here's a quick draft. It seems abour 10x faster (with 10% of your code, see my sig :D) than yours (same result). Modify as needed.

(This is not optimized for speed yet).

[altenbach]

Note that you can parallelize my FOR loop. (mine drops down to <2ms on my 16(32) core Xeon :D).

[JÞB]

oh darn!

the postage stamp version is coming!  I love it when I learn something. 

Saqueador

Just wait a bit.  Tim and Christian haven't fully optomized this yet and who knows what other ideas will show up.   (Cast to U8 compare to "0x30"  )

[Eric1977]

I used your example in the first sub VI and crossrulz example for similar code for comparing 0s in the first array and 1s in the second array in another SubVI.

Thnaks everyone again for your help.

Eric

[crossrulz]

---

@altenbach wrote:

Note that you can parallelize my FOR loop. (mine drops down to <2ms on my 16(32) core Xeon :D).

---

Wait a minute.  I thought you couldn't parallize a FOR loop that used shift registers.  When did that get fixed?  And now I'm wondering how it is working behind the scenes.  Something to go play with now.  Yep, not going to get any work done now.  Good thing it's Friday.

[altenbach]

---

@crossrulz wrote:

Wait a minute.  I thought you couldn't parallize a FOR loop that used shift registers.  When did that get fixed?  And now I'm wondering how it is working behind the scenes.  Something to go play with now.  Yep, not going to get any work done now.  Good thing it's Friday.

---

Certain code patterns are recognized as safe to parallelize. This is one of them. (else you'd get a broken arrow). Addition is commutative, so summing in a shift register does not depend on the order of the operations.

I don't know if there is a comprehensive list of such patterns. (some are listed here [link removed; page has been deprecated], look at figure 5, for example)

I even downconverted and tested in 2012 to make sure it still works. 😉

[YavannaKementári]

Wow, I learned something new today Altenbach. And here I've been just not parallelizing for loops with shift registers assuming that it doesn't work in all cases.