08-04-2017 07:47 AM
Hi all,
I was going through my code after I found it not to work properly.
I have narrowed it down to this example, is there a way around this?
I'm doing a comparison of test limits and the result is questionable..
Thanks
Erik
08-04-2017 07:59 AM - edited 08-04-2017 08:00 AM
Hi Erik,
oh those wonders of floating point numeric on computers…
You will find a lot of threads here in the forum on FP precision (topics like "never compare floats on equality" or "why is adding 0.1 ten times different than 1"). Read them, understand them.
Or go to Wikipedia…
08-04-2017 08:03 AM
Just so you know, this is a problem for ALL programming languages, not just LabVIEW. It is just inherent in attempting to use a finite number of bits to represent an infinitely accurate value.
08-04-2017 09:18 AM - edited 08-04-2017 09:20 AM
Oh boy another nickel. Here are just a few threads discussing this.
https://forums.ni.com/t5/LabVIEW/Comparison-Palatte-Equal-Not-working/td-p/1171217
http://forums.ni.com/t5/LabVIEW/1-00-gt-1-00-equals-FALSE/m-p/1099800
https://forums.ni.com/t5/LabWindows-CVI/Comparison-of-equal-doubles-fails/td-p/441277
http://forums.ni.com/t5/LabVIEW/comparison-of-2-double-values/td-p/2622041
http://forums.ni.com/t5/LabVIEW/DBL-comparison-errors/td-p/289693
Unofficial Forum Rules and Guidelines
Get going with G! - LabVIEW Wiki.
16 Part Blog on Automotive CAN bus. - Hooovahh - LabVIEW Overlord