spline fit behavior when input array size<7

[Jeraud]

Hi,

I'm having issues understanding the behavior of the cubic spline fit when the size of the input arrays is <7. I am using this vi to smoothen a series of data points but the output is not what I expect when the array size is less than 7. Attached is an example with an input of size=7, it works as expected. Delete the last value to make the size=6, and then the best fit output becomes very off. I don't understand that and it gives bad results for my smoothening VI. Any idea why does it do that and what could I do to fix my problem? Thank you very much!

Jerome

[Craig_]

Have you looked at the documentation for the function?

http://zone.ni.com/reference/en-XX/help/371361G-01/gmath/cubic_spline_fit/

You could manually calculate the values for the array size to verify that the VI is calculating correctly.

[johnsold]

Jerome,

As Craig pointed out, reading the detailed help for the VI may be useful.  Note that their example uses values of the balance parameter p of 0.9 and 0.995. This Vi is implemented in a DLL so you cannot see what it is doing behind the scenes.

It can be difficult to fit a smooth curve to data with big jumps like the point at t=4. Although if you change the Y-axis scale to 0.0003 to 0.0004, the fit looks very good regardless of settings or data points removed.

I modified your VI to allow easy exploration of the fit. It uses an event structure set to detect any changes in the inputs. Balance parameter values >0.98 but less than 1 seem to be closer to what you want. The behavior seems to be much less dependent on the details fo the data.

Lynn

[Jeraud]

Thank you very much for your input. Yes, I did go through the detailed help and it says that "Y must contain at least two points" so I expect the VI to work fine when the size of my array is less than 7 (and more than 1). One test that throws me is when setting the Balance Parameter to 0, the cubic spline fit should be equivalent to a linear fit. It is true when size > 6, but that doesn't look to be the case if you make 1 < size < 7. Am I facing a bug or do I do something wrong? Any input on how could I smoothen my data series when size<7?

Thanks!

[johnsold]

Jerome,

You are right. That looks like a bug.  It is certainly not producing a linear fit with balance = 0 and array size < 7.

Perhaps a polynomial fit might work better? With the order set to 1 you get a linear fit. With the order set to 3 the fit curves look similar to the spline fit on the original data, regardless of which element is deleted.

Lynn

[Jeraud]

Thank you for your reply Lynn and for confirming that this might be a bug (is there a way to report that to NI?).

The polynomial fit looks like a potential replacement when array size<7, but it is far from ideal as it tends to be too smooth and further away from the experimental data points when compared with the output of the spline fit set for low smoothness.

Cheers,

Jerome

[Craig_]

Hey Jerome,

Thanks for the info, based on this I have filed CAR 430579 noting the behaviour of cubic spline fit with arrays <7 elements.

[Darin.K]

How could I have missed this, this VI seems to be a poster child for what I love most about many of the math VIs.

1.  Bugs. Check.

2.  BD containing only a CLFN. Check.

3.  No reference to algorithm used.  Check.

4.  Incomplete functionality, unable to expand based on aforementioned CLFN.  Check

5.  Misleading help information.  Check.

Let's try to fix a few things while we are at it.

I use my own spline smoothing routine based on Reinsch's method, works like you would expect for all sizes.  Pts < 3 are special cases, 5 or six is certainly not an inherent problem.  7 does not seem to be a special number for any reason.

I really, really doubt that NI reinvented this wheel, so a reference to the implementation would be nice.  Plus it absolves you from most problems arising from confusing context help. 

Functionality: Let's ask ourselves why we are often using this function.  It is to build a smoothing function.  We have noisy data at a few points and would like a smooth curve to guide the eye on the display.  Piecewise linear plots just do not cut it sometime.  What is returned by the function?  The smoothed ordinate values only.  So now we are still plotting a piecewise linear function to smoother points.  Not ideal, and if you really are just smoothing a lot of data there are more efficient ways to do it.  Splines are great for making smooth interpolating functions.  What do we need?  At least the set of second derivatives found during the process.  With these we can reconstruct the full curve, not just the knots.

I find the statement that setting p=0 is the same as a linear fit a bit misleading.  Not quite true, minimizing only the curvature penalty (integral of second derivative squared) results in a family of lines.  You need a least-squares component to constrain it to the best-fit line.  Personally I use a penalty weighting for the integral term instead of a balance parameter (you can map them to each other).  By weighting just the integral term you get exact interpolation for weight=0 and a linear fit as the weight goes to infinity.  The L2 (least square) term is always around so the interpolating line is indeed the best linear fit.

The other hole in the documentation is the method for autotuning the balance parameter.  This is normally done using a secant or Newton's method search.  It requires you to specify an error tolerance and the algorithm finds the interpolant with the least amount of curvature which meets the error tolerance.  In the absence of any input, how exactly does the optimization work?  And, when I optimize the parameter using the secant method I return the optimized value as well.  It is good to know if it ran off of the rails.

Overall, the implementation is just a bit off.  It implements a smoothness function which is fairly sophisticated, but I bet few people use weights, and far fewer use smoothness.  But the values it returns (only the smoothed ordinates) are very basic and a waste of much of the work being done (inside the CLFN).

I could probably post a memory hog version of the method.  My real version, like most others out there, takes advantage of the fact that the beauty of cubic splines is that you get banded matrices so you can keep only a few diagonals instead of entire matrices (or calculate everything on the fly for minimal storage).  This means you need matrix routines for the compact representations and here is where a few short lines in Fortran can blow up into a rats nest of wires.  Using full matrices makes things easier to comprehend.

Good catch.

[johnsold]

Craig,

Thanks for generating and reporting the CAR.

Darin.K,

I agree with points 1..5.

Thanks for the list and for the comments which clarify and amplify the points in the list.  I rarely use splines, so your expertise and experience is valuable.  I had vague notions about some of those items but lacked the knowledge to explain my discomfort.

Lynn

[Darin.K]

Splines + Fitting = discomfort.  That is a good thing.  The limited functionality of this particular VI sort of drives you into the most disconcerting use of splines (fitting to data).  They are great for visual aides, if I have five points defining a contour on my intensity graph I would rather see a smooth blob than a polygon. 

And it was not on purpose (perhaps subconsciously) but that is pretty much an ordered list, from least to most annoying.