LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Garbage when using extended numbers

Hi there,
I am writing a LabVIEW VI that requires large decimal numbers. When I use an input value of say 819.2, the value that is actually entered in is something like 819.200012271. Why is this? I need the value to remain at exactly 819.2. Thanks!
0 Kudos
Message 1 of 2
(2,513 Views)
"Why"

There is no exact way to represent the ".2" part of the number in binary.

You can do .5 because it is 1/2.
You can do .25 because it is 1/4.

Exact fractions where the denomenator is a power of two can be represented. All others can only be aproximated.

Adding more digits (actually bit) to your number will get you a closer approximation, but never ".2".

Maybe this twisted example will illustrate.
In "trinary" number systems, you can have a value of ".1" base 3. This would be represented as a fraction as 1/3 EXACTLY! This can not be done in decimal, binary or any even numbered number system.

I hope this explains without getting into anything involving the greek alphabet.

Ben
Retired Senior Automation Systems Architect with Data Science Automation LabVIEW Champion Knight of NI and Prepper LinkedIn Profile YouTube Channel
0 Kudos
Message 2 of 2
(2,513 Views)