Cleaning a waveform.

[yoko911]

Hello there, i need some insigh on this project im coding.

In the atached files there is a .dat file, wich contains an array of 128000+/- numbers,

here are 2 images, one of the array on a graph (data)

and a code i use to extract the data from the file.(codigo)

Any ideas on how can i eliminate those big spikes?, i want to stay with the noisy signal just want to cancel any peaks, also this has to be dinamic, because this is data from a sensor, but other sensors have diferent offset or diferent amplitude, this is a bearing check and those peaks are imperfections, i want to eliminate them, then using an envelope detector (i have that, Hilbert-ish code), give the envelope some offset and then compare so i can see if the peak is within boundaries. 

i just need ideas on how to dinamically take those peaks, one idea i had in mind was slipiting the array in 10 pieces at least, then taking a percent of maximun values and replacing them with a Mean value, but that cropped the signal in diferents valleys and peaks (see below)

i've attached 30.dat wich is another sample, on a different offset. (they are zipped because forum wont let me upload them in .dat format)

Thank you in advance 🙂

[NeilR]

Hi

You could simply use the Y[i]=Clip{X[i]} function in the Signal Processing > Signal Operation palette.

With a value of 0.75 you get something like the 2nd waveform.

Cheers

Neil

[yoko911]

---

NeilR escribió:

Hi

You could simply use the Y[i]=Clip{X[i]} function in the Signal Processing > Signal Operation palette.

With a value of 0.75 you get something like the 2nd waveform.

Cheers

Neil

---

Thank you for your answer, the thing is, Signal varies in amplitude, so in some parts I could have something like this:

as you can see, the offset is around 3, if I clip it on 0.75 i would loose all the signal, im in need of somethin dinamic and not static as clip.

Thank you

[Fnavarrensis]

Hi yoko,

Maybe you would be interested in using the VI "Treshold detector" in combination with "Amplitude and Level" Express VI. Im attaching a quick example code where I´m using a Trimming index to remove the peaks. I would recommend indexes between .8 and 1, anything lower would be possibly removing valuable data of the signal. I tested it with both 30.dat and 7dat files and I think is working well. You can do the same for the negative (or low) numbers, in case your sensor has nagative peaks. Please let me know if this solution helps you or point you in the right direction. 

Have a nice day!

[yoko911]

---

Fnavarrensis escribió:

Hi yoko,

Maybe you would be interested in using the VI "Treshold detector" in combination with "Amplitude and Level" Express VI. Im attaching a quick example code where I´m using a Trimming index to remove the peaks. I would recommend indexes between .8 and 1, anything lower would be possibly removing valuable data of the signal. I tested it with both 30.dat and 7dat files and I think is working well. You can do the same for the negative (or low) numbers, in case your sensor has nagative peaks. Please let me know if this solution helps you or point you in the right direction. 

Have a nice day!

---

Thank you kind sir, but i have LV2011 i cant open that file 😞 so i cant open it right now 😞

[Fnavarrensis]

Hi yoko,

There you go, please let me know if it helps, cheers!

[Fnavarrensis]

Hi yoko,

There you go, please let me know if it helps, cheers!

[yoko911]

---

Fnavarrensis escribió:

Hi yoko,

There you go, please let me know if it helps, cheers!

---

Thank you, it was ok but i need something more dynamic, because i have pieces like the one below, using you code snippet, it doesnt behave the way it should, 

also keep in mind that some pieces are clean and with this method maybe clean waveforms will loose data, i've attached the above data.

Thank you 🙂

[Chris_12345]

Hi Yoko, what if you tried filtering out the spikes by extracting triplets of data points (points 0,1,2 then points 1,2,3 then points 2,3,4 then points 3,4,5 etc), examine each triplet, and see if the center point is a lot larger than the two points that are immediately before and after it. If so, replace that center value witht he mean of the two points that are immediately before and after it. This should give the dynamic behavior you are looking for. You can even save the spikes that you filter out if you want.

[Fnavarrensis]

Hi yoko,

Using the same code, made it a little bit more dynamic, now it will adjust the trimming index and decrease it a little more until it detects 3 (or whatever value entered in control) trimmed peaks, this means that the triming level will vary for every waveform. I checked with the 3 groups of data you posted and it looks pretty well. If you want it to be more severe, you can increase the number of trimmed points, with the risk of deleting some waveform data. As you can see, there´s plenty of modifications we can do on the code, so I recommend you to play with the similar fucntions if another specification should appear.

Hope this helps, cheers!