LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Number to Fractional String not using "round to even"?

Solved!
Go to solution

Number to Fractional String rounding question.png

For sanity check:

https://www.calculatestuff.com/math/rounding-numbers-calculator?number=1.055&round_to_places=2&round... 

Bill
CLD
(Mid-Level minion.)
My support system ensures that I don't look totally incompetent.
Proud to say that I've progressed beyond knowing just enough to be dangerous. I now know enough to know that I have no clue about anything at all.
Humble author of the CLAD Nugget.
0 Kudos
Message 1 of 8
(2,554 Views)
Solution
Accepted by billko

Betcha it's the same basic problem as this recent thread.

 

Even though you typed "1.055", and the display format is set to only show 1.055, try expanding to 20 digits of precision...

 

 

-Kevin P

CAUTION! New LabVIEW adopters -- it's too late for me, but you *can* save yourself. The new subscription policy for LabVIEW puts NI's hand in your wallet for the rest of your working life. Are you sure you're *that* dedicated to LabVIEW? (Summary of my reasons in this post, part of a voluminous thread of mostly complaints starting here).
0 Kudos
Message 2 of 8
(2,538 Views)

Still yet, proper rounding ignores everything right of the significant digit.  If I say to round to two places, you should look no further than three.

Bill
CLD
(Mid-Level minion.)
My support system ensures that I don't look totally incompetent.
Proud to say that I've progressed beyond knowing just enough to be dangerous. I now know enough to know that I have no clue about anything at all.
Humble author of the CLAD Nugget.
0 Kudos
Message 3 of 8
(2,528 Views)

@billko wrote:

Still yet, proper rounding ignores everything right of the significant digit.  If I say to round to two places, you should look no further than three.


e.g., 1.05509 should round to two digits the same as 1.05500.

Bill
CLD
(Mid-Level minion.)
My support system ensures that I don't look totally incompetent.
Proud to say that I've progressed beyond knowing just enough to be dangerous. I now know enough to know that I have no clue about anything at all.
Humble author of the CLAD Nugget.
0 Kudos
Message 4 of 8
(2,526 Views)

That's also why I made the bottom branch of the code.  It shows how it gets rounded if the same digits get treated as a floating point value with just one decimal place and rounded using the rounding primitive.

Bill
CLD
(Mid-Level minion.)
My support system ensures that I don't look totally incompetent.
Proud to say that I've progressed beyond knowing just enough to be dangerous. I now know enough to know that I have no clue about anything at all.
Humble author of the CLAD Nugget.
0 Kudos
Message 5 of 8
(2,519 Views)
Solution
Accepted by billko

@billko wrote:

@billko wrote:

Still yet, proper rounding ignores everything right of the significant digit.  If I say to round to two places, you should look no further than three.


e.g., 1.05509 should round to two digits the same as 1.05500.


But going to 20 dig. precision, the number is 1.05499999999999994, thus round down to 1.05

0 Kudos
Message 6 of 8
(2,513 Views)

@RavensFan wrote:

@billko wrote:

@billko wrote:

Still yet, proper rounding ignores everything right of the significant digit.  If I say to round to two places, you should look no further than three.


e.g., 1.05509 should round to two digits the same as 1.05500.


But going to 20 dig. precision, the number is 1.05499999999999994, thus round down to 1.05


Ouch - that's awesome.  I thought it was going to be something like 1.05500000000001

Bill
CLD
(Mid-Level minion.)
My support system ensures that I don't look totally incompetent.
Proud to say that I've progressed beyond knowing just enough to be dangerous. I now know enough to know that I have no clue about anything at all.
Humble author of the CLAD Nugget.
0 Kudos
Message 7 of 8
(2,507 Views)

@billko wrote:


Ouch - that's awesome.  I thought it was going to be something like 1.05500000000001


Sometimes those tails of the number will go that way.  Sometimes they are under.

Message 8 of 8
(2,501 Views)