LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

21 > 21 ?

Solved!
Go to solution

I'm having a program increment until it reaches or passes a value. I noticed it would increment once too many times. Used Highlight Execution and noticed that my Greater? was outputting true with two 21s. Both are doubles (64-bit precision). 

 

Incrementing by 0.25 leads 21=21, but when incrementing by 0.2, they are not considered equal and it will stop at 21.2. 

 

When incrementing to a height of 0.4 or lower, it will stop at the correct increment, but with a height of 0.6 and onwards and extra increment is always included. It stops correctly with 0.1 and 0.25 increments leaves me very confused. 

 

I attached the VI so y'all can try it out. I'm new to LabVIEW so let me know if there's a more efficient way to do something you see in my VI.

0 Kudos
Message 1 of 4
(1,875 Views)
Solution
Accepted by topic author MarkJonesATX

Expand your indicators to show 15, 16 decimal places.

 

I bet you are running into a problem with floating point numbers on computers where a number can not be exactly what you think it is.  A number that is obtained by math on other numbers, may not be exactly equal to what you think.

 

http://effbot.org/pyfaq/why-are-floating-point-calculations-so-inaccurate.htm

 

 

0 Kudos
Message 2 of 4
(1,865 Views)
Solution
Accepted by topic author MarkJonesATX

The short version is that any number with a fractional part that isn't exactly some number divided by a power of two can't be represented exactly by a PC.   So 0.25 works (1/4th) as does 0.5, and some that seem strange will work fine like 0.2890625‬ (37/128), but 0.1 and 0.4 and such won't work.

 

A common workaround is to either consider numbers equal if they are within some tolerance amount (0.000000001 or less, perhaps).  Another common one is to multiply your value by a few powers of 10 and then express it as integers.  So instead of going to 21, go to 21000, and always increment by whole numbers.

0 Kudos
Message 3 of 4
(1,846 Views)

Thanks guys! Its working now. Definitely something I need to keep in mind. 

0 Kudos
Message 4 of 4
(1,837 Views)