Coercion problems with double data type controls.

[J.Mamakos]

I'm pretty sure this has been discussed before, but I could find any topics that contained any solution of sorts.

I am observing a coercion problem when using a double data-type control to increment its value in 0.1 steps. I know that doubles cannot always be accurately represented in a binary system, but this particular calculation is puzzling me.

I have set up two side-by-side sections of the same code, one taking values from the control and the other making its own values.  There is a difference in the outputs. I want to be able to use the control, but to achieve the "Sim Data" output. Any ideas how I can force the issue?

The code sections laying side-by-side:

The results:

The left output is desired (I only put it in an array so we can easily see the history of outputs) but I need to use the manual control.

[J.Mamakos]

Even with this slight modfication, it doesn't work as I'd like (I also tried converting to U32 instead of using "round to nearest", but to no avail).

[johnsold]

The problem is, as you suspected, related to the representation of doubles in binary. 0.1 has an infinitely repeating representation in binary. So as you increment the 0.1 steps in the double control, the round-off error accumulates and eventually produces the results you see.  Change the display format for the indicators in the Man Data array to show 16-20 significant figures to see what I am talking about. Changing the display format similarly on the y control may be informative.

In the Sim Data calculation the input is always an integer.  So the round-off error does not accumulate.

Lynn

[GerdW]

Hi,

multiply your "y" values by 10, then round to nearest, then use the calculation for "sim data"...

As you noted before: DBL cannot represent fractions of 10 accurately!

[johnsold]

Make the y control an integer and use a double indicator to show the value of y as calculated in Man Data.

Lynn