Removing Peaks and Drifts

[simply_me]

Hello,

The attached vi is for post processing data from a data file. While finding the mean, STD, and etc... Is, relatively, straight forward, I'm looking to find the "true mean" which means the mean without considering particle drift (and without the jumps too if possible). I've tried using FFT and a high pass filter but it wasn't good, and I'm not sure that's the best approach....

What's the best way to do this?

Attached: vi, sample .txt file for processing, screen shot to clarify.  

Thank you,

[jareds]

Hi simply_me,

I don't know if I quite follow what you're trying to accomplish.  What is this data from?  What do you mean by particle drift and jumps?  Finally, what exactly do you mean by "true mean"?

[simply_me]

Hi Jareds,

This is an experimental data using particle analysis on droplets (~30um). 

See the new attached file to explain jump & drift. Basically, a drift is a slow change in the position of the drop (either z or y axis), and a jump is a sharp change in the location (~i.e. within 2-3 data points). I'm trying to exclude the drift data from the statistical analysis since it portrays an inaccurate picture on the data. I care about changes in position but not slow varying drifts.

I've added a high pass filter that seems to somewhat work, maybe it'll explain better....

I don't think that the filter setting that I have now is so reliable because some of the jumps are smaller than the original jumps (like the very last jump for example). The final goal is to be able to detect all jumps higher than 10 (or whatever the criteria will be) on the z axis and perform valid statistics on it.

Thank you, 

[aeastet]

Try this example

[simply_me]

Hi Tim,

Can you please save the vi in an older format?

Thank you,

[aeastet]

Here is 8.0

[johnsold]

From looking at your images it appears that you have several problems.

1. The raw data is digitized rather coarsely.  It appears that the LSB of the converter represents ~0.5 um and your signal is ~3-4 um peak to peak.  Not very good resolution.

2. Is the negative-going spike at sample ~11500 data or a jump?

3. The frequency of the drift overlaps with the frequency of the desired signal.  A filter cannot remove all of the drift without attenuating the desired signal.  At Samples 7500 and 13600 the drift component changes by about 2 um in one sample time.

I think you can eliminate most of the jumps with a threshold detector.  If |Z[i]| > |Z[i-1]| + zth, then replace Z[i] with NaN or the mean of Z[i-1] through Z[i-k].  The threshold, zth, could be about 3-4 um.

Can you amplify the signal before digitizing it?  This will improve the resolution.  If the offset is too large, a simple high pass filter with a cutoff "frequency" of one cycle per 1000-2000 samples would remove the offset and some of the drift without much effect on the desired signal.  You would need to keep the gain low enough that the Jumps do not saturate the amplifier or A/D converter.

Lynn

[simply_me · Accepted solution]

@ Tim, thank you for the modifications, I'll definitely use it to show the drift component. 

@ Lynn, thank you for the input. 1. I can't amplify the signal, and it's not desired in general. The data is collected in RT with CMOS 6.7um.

                                                2+3. The point at 11500 is a jump, and sure I agree with your claim at #3, but that's the best I have....

Is subtracting the drift graph from the original graph is a good enough approximation? (sure there is some noise, but I think it's pretty close?!)

See attached jpg.

Edit: Forgot to label graphs on attachment

[johnsold]

Only you (or your customer or boss or ...) can decide what is good enough.  That is often the toughest part of a project.  Since your desired signal and drift have overlapping frequency spectra, you cannot completely separate them with any filter. If you have enough knowledge about the underlying physical mechanisms which generate each component, you may be able to create a model which could be fit to the data to separate them.

Lynn

[simply_me]

Thank you both for your assistance.