How to find the "leveled out" area of a curve

[flycast]

I have a waveform that has a LOT of noise. It is a PSI sensor that is getting interference. I have done a lot to clean it up without losing the original path of the data. This is displayed in the top chart. I then get the derivatives of that curve and am using that to get the smooth area (the green). I was hoping that the derivatives would help be identify the "smooth, flattened" part of the curve. It turns out that the derivatives have a ripple pattern in it. That is the bottom chart. I have tried all kinds of curve fitting to try to smooth the derivative chart out but nothing seems to be working.

Can anyone here please suggest a technique?

[GerdW]

Hi flycast,

---

@flycast wrote:

I have a waveform that has a LOT of noise. It is a PSI sensor that is getting interference. I have done a lot to clean it up without losing the original path of the data. This is displayed in the top chart. I then get the derivatives of that curve and am using that to get the smooth area (the green). I was hoping that the derivatives would help be identify the "smooth, flattened" part of the curve. It turns out that the derivatives have a ripple pattern in it. That is the bottom chart. I have tried all kinds of curve fitting to try to smooth the derivative chart out but nothing seems to be working.

Can anyone here please suggest a technique?

---

Unfortunately you:

• forgot to attach a VI with real data embedded
• your two graphs don't use the same X scale
• your two graphs use different autoscale settings so we cannot compare the curves visually…

What is YOUR DEFINITION of a "smooth area"?

This definition will allow you (and us) to suggest an algorithm…

[altenbach]

We really don't know what you tried, so we cannot tell what's "not working". Do you have a mathematical model for the curve?

The derivative obviously oscillates around a slope of zero. Maybe all you need "linear fit ptbypt" with a reasonable history size and detect when the slope (absolute value) is below a certain threshold.

We have a lot of missing detail. Does the data arrive one point at a time and you need to detect the "leveled out" as data comes in or do you analyze the final dataset?

[Kevin_Price]

---

@altenbach wrote:

...Maybe all you need is "linear fit ptbypt" with a reasonable history size and detect when the slope is below a certain threshold.

+1

This can be a very effective technique for data as generally well-behaved as yours.  The history size acts as a kind of lowpass filter.  It should be big enough to help suppress your high-freq interference noise, but small enough to still show the true dynamics of the phenomenon you're measuring.

- Kevin P

[BowenM]

If it was me...

I'd just do a moving average filter on the original data with as big of a window as I could get away with before changing the shape of your curve. Then I'd take the derivative and just set a threshold below which the curve is considered to have reached the "leveled out" portion.

Edit: Its worth commenting that this is effectively the same thing that altenbach / Kevin_Price both suggested. Get a big enough average of data and then check the slope. Their method is probably easier to implement.

[altenbach]

---

@flycast wrote:

I have a waveform that has a LOT of noise. It is a PSI sensor that is getting interference. I have done a lot to clean it up without losing the original path of the data. This is displayed in the top chart. I then get the derivatives of that curve and am using that to get the smooth area (the green).

---

I have a hard time telling how your "smooth area" is defined. There is no real obvious feature in your data or derivative to point at that exact green starting point.

What exactly did you do to "clean it up"? Sometimes working on the raw data is better that fudging. For example why does your derivative plot name have "spline" in the name?

Also don't know what a PSI sensor is. Are you talking about pressure? How can you guarantee that the curve flattens out? Could it e.g. go down again due to some hardware problem?

Can you attach a simple VI containing typical raw data. Also give the exact specification how "flattened out" is defined.

[JÞB]

---

@flycast wrote:

I have a waveform that has a LOT of noise. It is a PSI sensor that is getting interference. I have done a lot to clean it up without losing the original path of the data. This is displayed in the top chart. I then get the derivatives of that curve and am using that to get the smooth area (the green). I was hoping that the derivatives would help be identify the "smooth, flattened" part of the curve. It turns out that the derivatives have a ripple pattern in it. That is the bottom chart. I have tried all kinds of curve fitting to try to smooth the derivative chart out but nothing seems to be working.

Can anyone here please suggest a technique?

---

How are those graphs related in time?  What are the vertical units?  What have you done to limit shock and vibration applied to the system?  Are the pressure fluctuations "Real," or introduced by someone walking around?  Is the medium gas, fluid or solid (other?)

[BertMcMahan]

FYI: I have used splines for smoothing data in the past, and they can cause ripples like the ones you're seeing if used improperly. Perhaps your input data isn't rippling at all.