Thresholding 1D array

[McMurray]

Hello everyone,

I have a relatively simple task that I'm banging my head to try and solve.  Basically, I have a 1D array of doubles, and I need to delete the outliers from this array.  I'm defining an outlier as 1.5 times the standard deviation of the array.  Any thoughts?  This should be simple, but the only solutions I can come up with involve Case Structures inside of For loops and multiple copies of my array.  I should think there would be an efficient way of doing this, but I just can't think of it.  Thanks for the help!

Matt

[smercurio_fc · Accepted solution]

Search, search, search: http://forums.ni.com/t5/LabVIEW/Search-and-Erase-Element-from-Array/m-p/763109#M349702

[altenbach]

Saverio already provided you with a good link. Please read it! 🙂

Still, please define what you mean by "remove". If you simply delete these outliers, you will change the indices of all other elements, thus you change the x information, which is now no longer linear.

Another option would be to replace the outliers with something else (e.g. the SD value of the array, the average of the immediate neightbors, etc) while keeping the array size (and X information!) constant.

Can you tell us a bit more about your application?

[McMurray]

Thanks for all the help you two!  I wonder why my search didn't bring that up...

As for an overview of the application, I'm working with a previously recorded ECG signal, which has numerous artifacts due to subject movement, etc.  I'm trying to cull out the extremely odd data based on the interval between each peak (called the R-R inteval).  For example, an interval of 0.15 seconds would be ridiculous, as would an interval of 40 seconds.  I have about 10 minutes of data recorded, so I'm basing the "culling" on the mean R-R interval and standard deviation.

I've adapted the code from the link smercurio suggested, but for some reason the array indexing isn't working.  The output is an array of the correct size, but composed entirely of the first number repeating.  I've attached the vi.  Thoughts?

[altenbach]

You use the wrong VI in the case structure. Replace "insert into array" with "replace array subset" and you should be all good. 😄

Still, since you change the size of the array, your timing will be off. Instead, you should do someting along the lines I mentioned above.

[McMurray]

Oh, I'm dumb.  Thanks.

I'm not too interested in the timing, just the R-R interval.  I don't mind that loss of data if it means my R-R estimates will be better.  Thanks.

[altenbach]

I don't understand why the estimates would be better if you remove points. With each deletion you are moving the data closer together, thus shrinking the intervals.

[McMurray]

I probably should have explained the 1D array a little better.  That might help explain why this is helpful.  The array is currently only composed of the intervals, and has no relationship to the order or time at which they were collected.  So, deleting a few extraneous points doesn't change the intervals themselves, just the mean and standard deviation since the outliers have been removed.

[altenbach]

Ah, ok. Now it makes sense. 🙂