Determine the number of digits after decimal.

[moderator1983]

Hi All,

I've tried and made a VI, which will determine the number of digits after decimal place.
Now if I choose to enter say "0.981" and run the VI, it returns 16 as the number of digits after decimal.

Now after selecting the "Display Format" of "Number In" I made following changes:

and now I found that "Number In" is actually equals to "0.98099999999999998300" and not "0.981".

Why is this happening?

[nyc_(is_out_of_here)]

---

@moderator1983 wrote:

and now I found that "Number In" is actually equals to "0.98099999999999998300" and not "0.981".

Why is this happening?

---

That is happening because you have the number of digits of precision set to 20.

[moderator1983]

Please go through the VI and query once again.

I set the number of digits of precision to 20, just to probe, what exactly problem is....!! 

[nyc_(is_out_of_here)]

I can't run your VIs. I only have LabVIEW 8.6

[Rod42]

Hello,

This effect has ben the subject of many threads, Search for rounding error, roundoff error, or similar terms.

The number 0.981 cannot be represented exactly in a floating point number format, such as DBL, so the number stored is an approximation - close to but not EXACTLY 0.981

When you repeatedly multiply by 10, and perform equality checks you are seeing the effects of this approximation. It is an inevitable part of the way non-integer numbers are stored.

It is common to get apparantly contadictory results when you perform equality checks on floats.

This does not just affect LabVIEW. You will see it in any language that operates on floating point numbers.

Rod.

[GuenterMueller]

IEEE 754-1985 defines the in-memory representation for floating point numbers. There should be lots of documents available to read out what happens.

[moderator1983]

Hmmnn Lot of replies, thank you everyone..

Now after I'm aware of why it is happening, can someone help me with actual problem:

"Determine the number of digits after decimal."

[GuenterMueller]

If you do not like Rod's idea: What do you expect?

[Taki1999]

If you want more control of the format for data entry, you could use a string control instead of a numeric.

As mentioned, any numeric control you use will end up with a IEEE representation.

If you just want to retrieve the precision you've selected for your numeric control, you can use a property node.

[Rod42]

Hint: One way to avoid checking for equality in a real comparison is to check for the difference being sufficiently close to zero. Remember that the difference can be positive or negative. You will need to determine for yourself how close to zero "sufficiently" is for your application.

Rod.