Question about precision?

[Bill Choy]

Hi

I have a problem about precision. The example program has a digital control which default number is 0.999. I set the precision of that control to 2 and the value become 1.00. After I mutiply this value with 10 and truncate it. I got the answer is 9. The program used 0.999 for calculation but I want to use 1.00. I can solve it by changing the number to string and then change back to number. It works but so stupid. Are there any ways to solve it?

Bill.

[storeyk]

As i only have LV 6.0 i cannot open your example. But looking at your explanation i assume you dont want to change the default. So why dont you round the result or the value 0.999 to the nearest integer? That will give you 10 or 1 respectively. This function can be found in the Functions>Numeric palette.

NB. Setting the precision only changes the display, the number behind it is still the same.

Kim

[Bill Choy]

Hi Kim

Round to nearest is not suitable in my case becasuse it round to an integer. I need to round to 2 decimal points. How to do that?
Thanks for your help.

Bill

[asimpleman]

"ctbiu" wrote in message
news:506500000005000000BC930000-1027480788000@exchange.ni.com...
> Hi Kim
>
> Round to nearest is not suitable in my case becasuse it round to an
> integer. I need to round to 2 decimal points. How to do that?
> Thanks for your help.

Multiply by 100, round to nearest, divide by 100. You can generalize it to
any N decimal places by using the power of 10 function.

[wha@atmel]

Hi,

I have played with your example program and found out, that your "Input.nr" is incorrect. You should change "format & precision" to 3 digits of precision - then you will already see your problem - the value is 0.999 and then rounded to 1.00 in your case.
In addition look to "data range". There is used the 0.999 as default value !!! change it to 1.00E-1.

I think that you have copied this control from somwhere and you copied it's limitations.

regards
wha

[Bill Choy]

Hi wha

I set 0.999 to the default value and 2 digits of precision because I have a program facing this problem. I need to display the value to user with 2 digits of precision but the input value is 3 digits of precision. Therefore I can't change the format & precision.

Bill.

[storeyk]

Adam, you took the words right out of my mouth!

*smiles*

Kim