04-29-2020 03:39 PM
I'm having a program increment until it reaches or passes a value. I noticed it would increment once too many times. Used Highlight Execution and noticed that my Greater? was outputting true with two 21s. Both are doubles (64-bit precision).
Incrementing by 0.25 leads 21=21, but when incrementing by 0.2, they are not considered equal and it will stop at 21.2.
When incrementing to a height of 0.4 or lower, it will stop at the correct increment, but with a height of 0.6 and onwards and extra increment is always included. It stops correctly with 0.1 and 0.25 increments leaves me very confused.
I attached the VI so y'all can try it out. I'm new to LabVIEW so let me know if there's a more efficient way to do something you see in my VI.
Solved! Go to Solution.
04-29-2020 03:49 PM
Expand your indicators to show 15, 16 decimal places.
I bet you are running into a problem with floating point numbers on computers where a number can not be exactly what you think it is. A number that is obtained by math on other numbers, may not be exactly equal to what you think.
http://effbot.org/pyfaq/why-are-floating-point-calculations-so-inaccurate.htm
04-29-2020 04:30 PM
The short version is that any number with a fractional part that isn't exactly some number divided by a power of two can't be represented exactly by a PC. So 0.25 works (1/4th) as does 0.5, and some that seem strange will work fine like 0.2890625 (37/128), but 0.1 and 0.4 and such won't work.
A common workaround is to either consider numbers equal if they are within some tolerance amount (0.000000001 or less, perhaps). Another common one is to multiply your value by a few powers of 10 and then express it as integers. So instead of going to 21, go to 21000, and always increment by whole numbers.
04-29-2020 04:37 PM
Thanks guys! Its working now. Definitely something I need to keep in mind.