ā01-13-2020 04:29 PM
For sanity check:
Solved! Go to Solution.
ā01-13-2020 04:41 PM
Betcha it's the same basic problem as this recent thread.
Even though you typed "1.055", and the display format is set to only show 1.055, try expanding to 20 digits of precision...
-Kevin P
ā01-13-2020 04:51 PM
Still yet, proper rounding ignores everything right of the significant digit. If I say to round to two places, you should look no further than three.
ā01-13-2020 04:55 PM
@billko wrote:
Still yet, proper rounding ignores everything right of the significant digit. If I say to round to two places, you should look no further than three.
e.g., 1.05509 should round to two digits the same as 1.05500.
ā01-13-2020 05:04 PM - edited ā01-13-2020 05:04 PM
That's also why I made the bottom branch of the code. It shows how it gets rounded if the same digits get treated as a floating point value with just one decimal place and rounded using the rounding primitive.
ā01-13-2020 05:12 PM
@billko wrote:
@billko wrote:
Still yet, proper rounding ignores everything right of the significant digit. If I say to round to two places, you should look no further than three.
e.g., 1.05509 should round to two digits the same as 1.05500.
But going to 20 dig. precision, the number is 1.05499999999999994, thus round down to 1.05
ā01-13-2020 05:17 PM
@RavensFan wrote:
@billko wrote:
@billko wrote:
Still yet, proper rounding ignores everything right of the significant digit. If I say to round to two places, you should look no further than three.
e.g., 1.05509 should round to two digits the same as 1.05500.
But going to 20 dig. precision, the number is 1.05499999999999994, thus round down to 1.05
Ouch - that's awesome. I thought it was going to be something like 1.05500000000001
ā01-13-2020 05:20 PM
@billko wrote:
Ouch - that's awesome. I thought it was going to be something like 1.05500000000001
Sometimes those tails of the number will go that way. Sometimes they are under.