Number Division (DBL) Error regarding Decimal Places

[re-fuse]

hey guys,

i've encountered a problem when i want to divide a number by a larger number.
for example: in the case of 10/1000 the result should be exactly 0.01,

but what i get in labview is 0.0100000000000000002.

so there must be clearly something wrong in the calculation in labview. doesnt matter if i choose dbl, sgl or even ext as data format.
and it is indepedent of the computer where the calculation is done.

what's the problem here? neither windows calculator nor excel show the same behaviour.

thanks in advance for any help.

[GerdW · Accepted solution]

Hi re-fuse,

welcome to the long list of threads regarding floating point precision issues!

Floating point numbers are inaccurate, you have to handle this in your programs - regardless of the programming language you use! (Read the first sentence of this article!)

And don't forget: NEVER compare floating point numbers for equality…

[Shane-C · Accepted solution]

For a more comprehensive overview of how floating point numbers work, see this document.  (or any of the threads mentioned).

http://download.ni.com/pub/devzone/tut/floating_point_in_labview.pdf