LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

extra decimal places in double

Hello,

 

I am trying to make a program to round a number to the nearest thousandths place. I am seeing extra decimal places, which I didn't add, shown in the input. The screenshot below shows what happens after I entered ".001" as the input and ran the program. This accumulation of digits at far out decimal places is causing me problems in my code.

 


Thanks

 

Round to Thousandths.JPG

0 Kudos
Message 1 of 14
(3,396 Views)

This is because of the finite representation of numbers in the computer. 0.001 is an infinitely repeaing fraction in binary. That means that you can never exactly round off a number to 3 decimal places in a binary computer.

 

If you really need finite representation, multiply your values by 1000 and convert to integers. However, when you divide by 1000 again you will get the same kind of extended fractions for your doubles.

 

This is true of any computer language unless it is doing BCD arithmetic.

 

Lynn

0 Kudos
Message 2 of 14
(3,387 Views)

Hmmm so I guess for any operation I just have to multiply by a power of ten to get all my decimal points on the correct side. Then convert to an integer.

0 Kudos
Message 3 of 14
(3,367 Views)

@tda*! wrote:

Hello,

 

This accumulation of digits at far out decimal places is causing me problems in my code.

 


Thanks

 

 


What kind of problems is it causing?  As pointed out by others, what you are seeing is an inherent problem in dealing with floating point numbers in a computer's binary system.  It generally isn't a problem unless you are trying to use the numbers in a way they weren't meant to be used.

 

Instead of trying to recreate a computer's way of representing decimal numbers, you might need to solve the problem that your implementation of those numbers is causing.

0 Kudos
Message 4 of 14
(3,350 Views)

@RavensFan wrote:

@tda*! wrote:

Hello,

 

This accumulation of digits at far out decimal places is causing me problems in my code.

 


Thanks

 

 


What kind of problems is it causing?  As pointed out by others, what you are seeing is an inherent problem in dealing with floating point numbers in a computer's binary system.  It generally isn't a problem unless you are trying to use the numbers in a way they weren't meant to be used.

 

Instead of trying to recreate a computer's way of representing decimal numbers, you might need to solve the problem that your implementation of those numbers is causing.


Ugh, working with FP numbers is so tricky because comparing numbers is never a straightforward task.  At least, it's tricky for me.

Bill
CLD
(Mid-Level minion.)
My support system ensures that I don't look totally incompetent.
Proud to say that I've progressed beyond knowing just enough to be dangerous. I now know enough to know that I have no clue about anything at all.
Humble author of the CLAD Nugget.
0 Kudos
Message 5 of 14
(3,327 Views)

So here is an example or how I'm getting hung up. I round .10003 to .1. Then I subtract it from .6. I try to determine if the difference is divisible by .1 but the numbers don't come out in agreement.

 

program 2.JPG

0 Kudos
Message 6 of 14
(3,316 Views)

@tda*! wrote:

So here is an example or how I'm getting hung up. I round .10003 to .1. Then I subtract it from .6. I try to determine if the difference is divisible by .1 but the numbers don't come out in agreement.

 

program 2.JPG


If you round something to the nearest .1 and subtract it from .6, won't it always be dvisible by .1?  Or is this just an exercise to show us your problem?

Bill
CLD
(Mid-Level minion.)
My support system ensures that I don't look totally incompetent.
Proud to say that I've progressed beyond knowing just enough to be dangerous. I now know enough to know that I have no clue about anything at all.
Humble author of the CLAD Nugget.
0 Kudos
Message 7 of 14
(3,310 Views)

@billko wrote:

@tda*! wrote:

So here is an example or how I'm getting hung up. I round .10003 to .1. Then I subtract it from .6. I try to determine if the difference is divisible by .1 but the numbers don't come out in agreement.

 

 


If you round something to the nearest .1 and subtract it from .6, won't it always be dvisible by .1?  Or is this just an exercise to show us your problem?


Not in Floating Point.  since there is no IEEE 754 value that exacly represents 0.1 Multiply by 1000, 10000 (or whaterver you want for precision) convert to integer and do the comparison with integers.  You can cast back to float after the comparision


"Should be" isn't "Is" -Jay
0 Kudos
Message 8 of 14
(3,305 Views)

0.1 cannot be exactly represented in a binary format.  That is the danger of floating point numbers.  You will not have exact representation in the factional part unless it is a power of 1/2.

 

To see this, just put a numeric control down on the front panel, change the display to show something like 30 decimal places, and type in 0.1.  You will notice that your 0.1 is not truely 0.1.  This is a problem in all computer languages.


GCentral
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
"Not that we are sufficient in ourselves to claim anything as coming from us, but our sufficiency is from God" - 2 Corinthians 3:5
0 Kudos
Message 9 of 14
(3,304 Views)

@JÞB wrote:

@billko wrote:

@tda*! wrote:

So here is an example or how I'm getting hung up. I round .10003 to .1. Then I subtract it from .6. I try to determine if the difference is divisible by .1 but the numbers don't come out in agreement.

 

 


If you round something to the nearest .1 and subtract it from .6, won't it always be dvisible by .1?  Or is this just an exercise to show us your problem?


Not in Floating Point.  since there is no IEEE 754 value that exacly represents 0.1 Multiply by 1000, 10000 (or whaterver you want for precision) convert to integer and do the comparison with integers.  You can cast back to float after the comparision


I didn't represent myself clearly.  I was wondering about the reasoning behind doing this, not the mechanics of doing it.  🙂

Bill
CLD
(Mid-Level minion.)
My support system ensures that I don't look totally incompetent.
Proud to say that I've progressed beyond knowing just enough to be dangerous. I now know enough to know that I have no clue about anything at all.
Humble author of the CLAD Nugget.
0 Kudos
Message 10 of 14
(3,300 Views)