x, y data versus curve fit of data

[Bob_Schor]

I think I now understand what your data represent, but I'm not entirely sure I understand what you want to do.  That is to say, what is your hypothesis?  Are you looking for points where the two instruments "don't agree" with each other (suggesting one of them had a "bad reading")?  Are you looking for "weird points" (corresponding, perhaps, to a Full Moon) that might affect both instruments?  I must say, your "blob" looks pretty compact, to me ...

BS

[erfigge]

Well, the values actually are related.  One instrument is pressure, one is flow and at the given point in time a data point is taken the values of both of those instruments are recorded.  So we have two instruments values for that one data point.  They are related in the sense that the machine we are monitoring has a pressure versus flow relationship.  It will generally follow a curve like the black line.  For the example I showed, the light blue blob is sort of small, normally there would be lots more points further down along the black line.  

For discussions sake here lets say the other colored points were actually light blue points (ignore the lines between those points).  There is obviously something wrong with those points as they don't follow the general relationship as shown by the rest of the major group of light blue points.  I'm trying to find a method to kick those points out automatically, or at least if a user says I want to kick everything out that isn't within 90% of the rest of the points.  Then the user might say I want to kick everything out that isn't within 80% of the rest of the points and so on.  

My hypothesis is I can use a curve fit vi and generate the black line through the data points, but then I'm struggling to figure out how to mathematically bounce those points out.

Thanks for hanging with me while I'm attempting not very successfully to describe my situation and question.

Eric.

[johnsold]

Eric,

You need to be very careful when rejecting data point merely on the basis that "it does not look right" to a user. It becomes very easy for the user to learn to pick the points which make the remaining data look very good. In many areas this is called "cheating."

It is not unreasonable to reject points which are truly outliers, but the rejection process must be based on criteria that come from knowledge of the underlying principles of operation of the system.  However, if you do not know what is wrong with those points, you should be very careful about removing them. They might be trying to tell you that your system does not work as you expect it to. For example an intermittent leak might cause such behavior.

What causes the outliers in your system? Electrical noise from the motor controller which drives the pump/fan? Cavitation? How well do you know the relationship between pressure and flow for your system? Do you have an equation (ideally derived from Navier-Stokes or something) or just empirical curves?

If you fit all the data to the equation, do you get a curve like the black line but with an offset, a change in slope, or a change in curvature? What do such changes tell you about the process? Can you use that information to define an algorithm to determine which points are true outliers?

Lynn

[comrade]

Alright, if flow and pressure usually follow some kind of relationship then you should be able to find a curve fit, although from the blue light point blob in your plot I would never guess the black line to be this curve. However, to find the shortest distance between a point to a curve, try this one here:

Shortest Distance From a Curve To A Point