Removing Empty strings from 1d array

[VicEngineer]

Hello,

I'm looking to remove empty string data from a 1D array in a RT labview project running on a cRio-9068.

The program is running multiple loops simultaneously and I need to parse a 30-100 thousand array entries every second. The length of the arrays is variable based on the captured asynchronous serial data from 1 serial port running at 921600 Baud, and 3 at 115200 Baud, each has a separate 1D string array I can parse in parallel or I can parse the 1D array once they have all been combined into a single array.

My assumption is that the performance would be higher using 4 parallel loops parsing each array before it is combined.

My question is does anyone know the performance difference between the three following options? I've seen recommendation's for the OpenG option but no one has elaborated why its the preferred option, or to the performance differences between the options.

I'm using Labview 2014 with the most recent service pack.

[altenbach]

I am not sure what the OpenG version does (why don't you look at the code? :D), but you definitely want to do the bulk of the operation "in place", immediately eliminating the version using "delete from array" and the version using "built array". There are plenty of examples in the forum how to do it right (see e.g. the bottom code here).

 In addition, both of your examples are highly flawed.

Topmost: Wiring N as the array size is incorrect unless all elements are empty. You need to stop the loop once the search returns -1.

Middle: Your shift register needs to be initialized, else its data grows without bounds after successive runs during the same session.

(Also don't forget the conditional tunnel, resulting in very simple code. Need to benchmark!)

Can't you reject empty strings right after reading, and before building the arrays?

[Bob_Schor]

My prediction is going to be None of the Above, but I'm doing a test right now ...

Bob Schor

[Bob_Schor]

OK, I didn't test all of them, but I did test your first "delete" method and Open G.  My test case was 10,000 strings of length 0 through 9 (uniformly distributed, so about 10% would be length 0), and I fed the same array to all four cases.  All four returned 9028 strings (so about 10% were removed).

Here are the results:

   Your Delete method: 381 msec

   OpenG:  4.3 msec

   My "improvement" on your Delete Method (see below):  4.2 msec

   My "best" method:  1.8 msec

Here are the two "winners":

 and 

The first improves on the "Find and delete" method by starting from where it left off and quitting when done.  The second uses the Conditional Index to simply "rebuild" the array without the blanks, and wins.

Bob Schor

[altenbach]

You also need to look at the O scaling. Try with 100k elements. 😮 

[VicEngineer]

---

altenbach wrote:

Can't you reject empty strings right after reading, and before building the arrays?

---

The serial data is captured in the FPGA as U8 bytes and passed via DMA FIFO to the RT target on the cRio-9068.

My problem is that when im parsing the the U8 byte array looking for line termination (Line feed, Carriarge return, etc...) I have a false case in my code where the the wire is using the default tunnel assignment. Essentially it is adding in the empty string cases which I want removed from the final 1D string Array.

I'll post a code snippet later this work computer does not have Labview installed so I cannot open and screen shot any of the code.

Thank you for the suggestions and replies, very helpful.

And yes I am concerned about the O scaling, the data rates for the serial busses will likely be pushed to high baud rates which will make the CPU usage for these loops even more critical.

[Bob_Schor]

If I run the same test with 100,000 strings, 10% empty, the discrepancy between Bad and Good widens.

Delete -- 38.04 sec

Modified Delete (BS) -- 0.70 sec

OpenG -- 0.99 sec

BS Best -- 0.02 sec

I'm almost afraid to try 1,000,000 -- the Delete method seems to go as the square of the number of points ...

BS

[altenbach]

What is "BS Best"?

[Bob_Schor]

Wow.  At 1,000,000, the times continue to grow (of course):

Delete: 226 sec

Fast Delete (BS) : 141 sec

OpenG:  140 sec (oh, well)

Best BS: 0.2 sec

All loops left 899,848 strings after the 10% removal.

BS

[Bob_Schor]

Sorry, didn't notice the question -- BS Best is the last "Conditional-reassembly" example (you suggested it, as well, Christian ...).

BS