Multiply floating point rounds

[Wes_OH]

Why is it that when I use the Multiply function with two floating point numbers, that it rounds off the result?

I have the Format/Precision set to 3 decimals on the indicator.

The inputs are both doubles.

Using a probe before the indicator shows that the result is rounded by the mulitply function itself.

[GerdW]

Hi Wes,

no, LV doesn't round on it's own:

Using a (standard) probe isn't the best when trying to check rounding - you will get only limited precision. Use a custom probe or an indicator instead!

Message Edited by GerdW on 02-05-2007 04:55 PM

[altenbach]

---

@Wes_OH wrote:

I have the Format/Precision set to 3 decimals on the indicator.

---

NO! 😮

Your indicator is set to six digits of precision, thus shows only six significant digits and that's exactly what you get (324975). If you would set ot to 3 signiificant digits, it would display as (325000).

If you want to show a certain number of digits after the decimal point, you need to set the "precision type" to "Digits of Precision", not "significant digits", as you have it set now. Try it! 🙂

Also don't trust any probes or indicators, they never "round", i.e. never change the data. The displayed precision is just cosmetic and does NOT change the underlying data that is carried in the wire, which is always full precision. If you want a probe with 10 decimal digits, create a custom probe. 😉

If you want to round, you need to do it in code.

Message Edited by altenbach on 02-05-2007 08:37 AM

[Wes_OH]

Got it.

Thanks guys!