Filtering Noise from Measured Data

[lucasphee]

Hello,

I have some data on a current waveform that I measured using an oscilloscope.  That data was saved as a csv file, and in Labview, I have extracted the data for the waveform and put it into a 1D array.  The data that I measured was a little bit noisy and to do the analysis that I want to do on the data, I need to remove some of that noise, most importantly, some spikes that may have occurred.  Is there a way that I am able to do this using the data in my array?

Thanks

[PsyenceFact]

Hi lucasphee,

There are a number of filters on the Signal Processing - Filters palette which may suit your needs.  Depends on the frequency content of your signal, type of noise, and so on.  There's also Smoothing Filter Coefficients.vi on the Advanced IIR Filtering sub-palette which may be more relevant to your requirements.  AFAIK all the filters can be applied to a 1D array of floating point data.

Andy

[altenbach]

Noise and spikes are two very different things and might need separate treatment. How much spectral overlap is there between data and noise? What kind of analysis follows and how does noise influence the result?

Can you show us some typical data?

[lucasphee]

What I really need to eliminate is spikes.  However, if I am unable to remove them, even making them the same height as the peaks of the waveform would be helpful too.  My goal is to send these waveforms through a function generator to generate a current in a device.  However, I need to be able to control the RMS voltage of the function generator output with just one control and different waveforms with large spikes make changing the amplitude in order to change the RMS voltage difficult.  I have attached a few waveform examples.  

[paul_cardinale]

I think you're misusing the word "noise". Noise is any unwanted content.  Spikes are usually noise.

[altenbach]

Noise is typically continuous and has a characteristic frequency distribution (white, pink, 1/f, etc.) that depends on the system. Spikes are localized to very narrow randomly spaced sections, often single points, so they are more like artifacts or glitches. Do you know the origin of the spikes?

[PsyenceFact]

Whilst giving due consideration to the comments by altenbach and paul_cardinale (always give due consideration to comments by altenbach), if you're not too concerned about the source of the spikes and just want to reduce their presence in the data try passing your data through a high-pass Butterworth filter.  I can't suggest what the the cut-off frequency and order should be as you haven't specified the sampling frequency.  Just looking at the example graphs suggests that the "signal" part of the data is at higher frequencies than the "noise" part.  If you look at the data in the frequency domain using something like FFT Power Spectrum and PSD.vi it should give you an idea if this is a viable approach for your purposes and if so, where the cut-off should be located.

Andy

[altenbach]

It would be great to actually have a few datasets instead of just pictures. This way we could play around and test ideas. I might start with a running median filter, for example.

[paul_cardinale]

---

@altenbach wrote:

It would be great to actually have a few datasets instead of just pictures. This way we could play around and test ideas. I might start with a running median filter, for example.

---

A running median filter would be a great candidate for an XNode.

[paul_cardinale]

---

@altenbach wrote:

Noise is typically continuous and has a characteristic frequency distribution (white, pink, 1/f, etc.) that depends on the system. Spikes are localized to very narrow randomly spaced sections, often single points, so they are more like artifacts or glitches. Do you know the origin of the spikes?

---

While broadband noise may be more common (I don't know that it is), it is not the only kind of noise.  Noise is any stuff in the signal that you don't want.