Fract String to Number converts ”700,5” to 700,49999999

[Terje]

I found that for some numbers the Fract String to Number vi creates a small rounding error. As an example “700,5” converts to 700,49999999. (Se attached image and vi). Is there a way to fix this?

To see this error you have to adjust the number of digits to display in the Display format property tab. Or you can try using the round to nearest vi which will round to 700 and not 701. (This was how I found the error:) 

I’m using Labview 2009.

Terje

Message Edited by Terje on 02-08-2010 05:00 AM

Message Edited by Terje on 02-08-2010 05:01 AM

Message Edited by Terje on 02-08-2010 05:06 AM

[GerdW]

Hi Terje,

one comment on your problem (don't have LV at hand to check the problem):

700.5 will always be rounded to 700! LV uses rounding to next even number...

[Norbert_B · Accepted solution]

Terje,

this is expected behavior since a double (64 bit) is not capable to express 700.5.

You can check this by simply typing "700.5" in a numeric control and setting the display format for increased number of digits.

Since LV does not have any difference in regard to the numeric handling than other programming languages, i would expect the same behavior in other environments as well......

hope this helps,

Norbert

[Intaris]

Yes,

this is standard in almost all languges I can think of.

Shane.

[GerdW]

Hi Norbert,

to express 700.5 you need 10 bits for the mantissa (2^9+...+2^-1). Thus even a SGL should be capable to hold "700.5" with needed precision!

Just found a link to needed documentation...

Message Edited by GerdW on 02-08-2010 01:27 PM

[GerdW]

Just a correction: "700.5" needs 11 bits mantissa

[Ray.R]

LOL!  It's amusing to see how often this misinterpretation happens.

Gerd,  11 bits? ... interesting...  Definitely a trivia question.

[Norbert_B]

Gerd,

i just searched in the web a little bit in order to shed some light on the issue. Sadly, i found no real "solution", but something interesting:

Due to Wikipedia (See in chapter "Character Representation"), the rounding for floating point numbers is correct with 17 decimal digits for binary64 values (which LV-doubles are).

I previously found out, that the change from 770.5 to 700.4999 is done if the number of significant bits in the display option is set to something higher than 17 (so 18 or above).

I find this is a fascinating coincidence......

Norbert

Message Edited by Norbert B on 02-08-2010 07:52 AM

[Norbert_B]

Please note that there are several numbers with such "issues" like 2.1, 5.2, 7.3 and so on. In fact, there are nearly an unlimited number of numbers creating this behavior.

And, as i already stated, this is something common to computer based numeric representations.....

Norbert

[GerdW]

Hi Norbert,

there's a huge difference between 7.3 and 700.5: whereas 700.5 is representable in binary you cannot represent 7.3 as limited precision binary.

700.5 needs just 11 bits for the mantissa (1.0101111001b*2^9, easy enough even for SGL), but you need an unlimited number of bits for 7.3 (recurring binary fraction).

But then we should also ask for reason of displaying a precision of 17+ decimals. Usually you only have a measurement resolution of upto 7 digits...