02-08-2010 04:58 AM - edited 02-08-2010 05:06 AM
I found that for some numbers the Fract String to Number vi creates a small rounding error. As an example “700,5” converts to 700,49999999. (Se attached image and vi). Is there a way to fix this?
To see this error you have to adjust the number of digits to display in the Display format property tab. Or you can try using the round to nearest vi which will round to 700 and not 701. (This was how I found the error:)
I’m using Labview 2009.
Terje
Solved! Go to Solution.
02-08-2010 05:56 AM
02-08-2010 06:21 AM
Terje,
this is expected behavior since a double (64 bit) is not capable to express 700.5.
You can check this by simply typing "700.5" in a numeric control and setting the display format for increased number of digits.
Since LV does not have any difference in regard to the numeric handling than other programming languages, i would expect the same behavior in other environments as well......
hope this helps,
Norbert
02-08-2010 06:23 AM
Yes,
this is standard in almost all languges I can think of.
Shane.
02-08-2010 06:24 AM - edited 02-08-2010 06:27 AM
Hi Norbert,
to express 700.5 you need 10 bits for the mantissa (2^9+...+2^-1). Thus even a SGL should be capable to hold "700.5" with needed precision!
Just found a link to needed documentation...
02-08-2010 06:43 AM
02-08-2010 07:09 AM
LOL! It's amusing to see how often this misinterpretation happens.
Gerd, 11 bits? ... interesting... Definitely a trivia question.
02-08-2010 07:51 AM - edited 02-08-2010 07:52 AM
Gerd,
i just searched in the web a little bit in order to shed some light on the issue. Sadly, i found no real "solution", but something interesting:
Due to Wikipedia (See in chapter "Character Representation"), the rounding for floating point numbers is correct with 17 decimal digits for binary64 values (which LV-doubles are).
I previously found out, that the change from 770.5 to 700.4999 is done if the number of significant bits in the display option is set to something higher than 17 (so 18 or above).
I find this is a fascinating coincidence......
Norbert
02-08-2010 08:34 AM
Please note that there are several numbers with such "issues" like 2.1, 5.2, 7.3 and so on. In fact, there are nearly an unlimited number of numbers creating this behavior.
And, as i already stated, this is something common to computer based numeric representations.....
Norbert
02-08-2010 09:38 AM
Hi Norbert,
there's a huge difference between 7.3 and 700.5: whereas 700.5 is representable in binary you cannot represent 7.3 as limited precision binary.
700.5 needs just 11 bits for the mantissa (1.0101111001b*2^9, easy enough even for SGL), but you need an unlimited number of bits for 7.3 (recurring binary fraction).
But then we should also ask for reason of displaying a precision of 17+ decimals. Usually you only have a measurement resolution of upto 7 digits...