remove empty elements of array taking soo much time

[mabaig]

Hi guys 

I need help of your wise knowledge and experience

I am removing empty elements from an 1D array but its taking 11 sec(because of so many elements yet,). 

Is there any way to reduce its execution time. Can try different methods of removing empty string if it reduces its execution time

*Note : I have created this 1D input array from Intel Hex file of 1776 lines and it is created in 64ms.

Thanks and Regards 

Baig 

[Yamaeda · Accepted solution]

Don't use Remove from array.

Simply use a conditional array at the loop border and let it build a new array with all non empty rows. That way you're only making one memory allocation (or a few) instead of a full array copy each iteration.

/Y

[crossrulz · Accepted solution]

Two things that your VI is doing that is really bad.

1. Delete From Array causes a memory allocation each time it is called.  So for your example, I see 129 memory allocations of large arrays.  That is very expensive in both time and memory.

2. Each time you find an empty string, you start searching from the beginning again.  This means you hit that first element at least 128 times.  Again, a waste of time.

As Yamaeda pointed out, the more efficient option is to use Index Array tunnels and make the output tunnel be conditional.  Only hitting each element once and only a total of 2 possible memory allocations (the output tunnel defaults to the full array in size and then is stripped down due to the conditional).

[mabaig]

Awsome guys.... That was silly of me,lol.  

Thanks alot. Help is truely  appriciated.

sorry for more trouble, but I have one more question, Is it ok to use string subset frequently??   have used string subset like nearly 10 times in one VI (in a state machine where in each state I am seperating string at different offset address) ... Just need an advice!!

thanks and regards

Baig

[altenbach]

---

@mabaig wrote:

*Note : I have created this 1D input array from Intel Hex file of 1776 lines and it is created in 64ms.

---

Depending how you are "Creating" the 1D output from the file (I don't even know what a "HEX file" is!), you could probably wrap the filtering together with the parsing so the array is created without empty elements from the beginning. No need to create a large array just to throw most of it away a nanosecond later. 😄 1776 lines is not much.

Can you show us the rest of the code and a typical file?

[Yamaeda]

---

@mabaig wrote:

Awsome guys.... That was silly of me,lol.  

Thanks alot. Help is truely  appriciated.

sorry for more trouble, but I have one more question, Is it ok to use string subset frequently??   have used string subset like nearly 10 times in one VI (in a state machine where in each state I am seperating string at different offset address) ... Just need an advice!!

---

It's not easy to know that Remove from Array is such an expensive operation, but now you know. 🙂

I don't think String subset is that bad, you should be fine, but you can always create a performance test to compare a couple of solutions if you're wondering. 🙂

I remember a similar problem to yours, but i think i was using Match pattern or something, and the difference in speed was surprising!

/Y

[mabaig]

Thanks for response, You guys are always reliable. 🙂

I have attached the hex file sample.

File I am dealing with has some 60K lines.

And I had to do parsing of this 60k hex file into 1D array of two digit in each element. So I first made an 1D array of each line then I processed each line into again 1D array of two digits (Note: it has an specific order to arrange these 2 digit 1D array depending on the offset address of the string line which is why i had to use string subset so many times), then I created 2D array then again i converted this 2D array in to 1D array using resize array function where I ended up with some empty elements in this final array(Empty elements are due to unequal size of each line)  That's how I needed to clear empty elements.

Hope I was able to give you a clear idea of why i did what i did.

Thanks  and Regards 

Baig

[altenbach]

---

@mabaig wrote:

I have attached the hex file sample.

---

No, this is just an image of a hex file that is useless for us. Please attach the actual file (you can put in inside a zip file, together with the VI you are using to read it!).

We cannot debug images, for example we cannot see unprintable characters and delimiters. We can't see 60k characters and we also don't know how the fields are defined. Which part of the file shown would result in an empty array element?

[billko]

---

@altenbach wrote:

---

@mabaig wrote:

I have attached the hex file sample.

---

No, this is just an image of a hex file that is useless for us. Please attach the actual file (you can put in inside a zip file, together with the VI you are using to read it!).

We cannot debug images, for example we cannot see unprintable characters and delimiters. We can't see 60k characters and we also don't know how the fields are defined. Which part of the file shown would result in an empty array element?

---

Not to mention that I really don't feel like recreating all that text.

[altenbach]

Going back to my earlier suggestion, you should be able to parse that file from the beginning so no empty strings are ever being generated. It just needs to be a tiny little bit smarter with a little more (or possibly much less!) code. That's why we also need your VI that we can assume works correctly but is just too slow.