Strange math-problem with LabVIEW 7.1

[Espelkamper]

Hello!

I am using LabVIEW 7.1 and I came to this strange math-problem. Could it be that it is a problem of LabVIEW 7.1?

Ok try this with LabVIEW 7.1:

open a new VI

place a subtract-VI (pallet numerical) on the block diagram and try to subtract 0.2 from 1.6

place a less-VI (pallet compare) on the block diagram and wire a numerical constant 1.4 to input x

wire the result of the subtraction to input y of the comparison

place an LED to the output of the comparison

and start this.

What does it say?

The LED should turn on and this means here that 1.4 would be less than 1.4 .

But I would have expected that 1.4 is not less than 1.4 .

This works also if you add or subtract upto 0.2 to/from the values 1.4 and 1.6 (or equivalent with 0.2) .

If you change the inputs of the comparison then the result is as expected and 1.4 is reported not to be less than 1.4 .

If you compare two constants with the same value 1.4 you will not get the same strange result.

Does anybody have an explanation for this?

Espelkamper

[Will.D]

The reason for this is that the number you're comparing may not be exactly 1.4.  It might be 1.40000001 or something like that.  Great care needs to be taken when using comparisons with non-integers, as they may not be exactly what you think they should (i.e. "equal to 0" is very bad if you're trying to equate non-integers).

I knew at one time why this was, but for some reason it's not coming to me now.  It has to do with the resolution of the datatype, and the way the computer handles it I think, but I'm sorry I can't give you anything better than that...

I know there are many people around here that could though....hint hint.

[johnsold]

This is the comon result of the finite bianry representation of decimal numbers. If you look at the numbers with the maximum available resolution you will find that the 0.2 is really 0.19999999999.. or something similar. For this reason it is never a good idea to do an equality test with non-integer representations of numbers.

This comes up freqeuntly. Search the archives and the LV documentation for "numerical representation."

Lynn

[Will.D]

Thanks Lynn.  I knew it was something that'd make me look stupid.

It's been a long week...

[Espelkamper]

@ Novatron & johnsold

Thank you for your comments, they are interesting for understanding.

Maybe you could help me a little bit further.

I came to this problem through a check if a value (dbl) is in range (or not).

In this case I had an input value (dbl) of 1.40 ,

a reference value (dbl) of 1.60

and a tolerance (dbl) of 12.50 (%),

meaning is 1.40 in between 1.40 and 1.80 .

The check consisted of multiply, divide, add, subtract and comparison (greater or less).

The result was that 1.40 was out of range (1.40 to 1.80) and that was a surprize for me.

Does anybody have a better idea of how to solve this task without such a surprizing effect?

Thank you!

Espelkamper

[chrisger]

hi there,

when dealing with physical values you have a limited physical resolution, so i'd truncate the numbers to that resolution (lets say 2-4 digits). because the resolution of the fractional representation is much higher than your physical resolution you won't have that problems any more.

[GerdW]

Hello,

For floating point math: Rounding to multiples of 0.1 is problematic as those numbers are infinite fractions in base2 representation...
So maybe you should change your numbers to integers using fixed point math. If all your input values are in the range of 1.4 to 1.8 you can multiply by 10000 and round to next integer. Then do your comparison on that integer (there's also an 'InRange' function in the comparison palette).