LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Rounding and Precision

I came across this strange problem while checking alarm limits.

I've provided a VI to demonstrate the issue.

I have digital numeric control. The precision is two decimal places (0.00) and it incremented by (0.1). The default value of this control is 1.57. If you change the value of this control to some other number (1.8 for example) and then try to change the value back to 1.57, the value is rounded up to 1.60.
Now if you try to change the value to 1.55, the value is rounded down 1.50.

Can someone explain why:
1. the values of #.#1 thru #.#9 are rounded to the "nearest" 10th when they are typed in.
2. the value #.#5 is rounded down, when it is traditionally rounded up.
0 Kudos
Message 1 of 4
(2,843 Views)
1. Is because you have the increment set to 0.1 Coerce to Nearest. Apparently if LV sees the data change from keyboard input it coerces to the nearest increment value. Change the "If Value is Out of Range" setting to "Ignore" and you shouldn't have a problem.

2. I don't know.

Brian
Message 2 of 4
(2,843 Views)
"Brian Vibert" wrote in message
news:50650000000500000024540000-1007855737000@exchange.ni.com...

[rounding down at 0.5 rather than up]

> 2. I don't know.

Choice of the coders; it's no big deal. If the value is below 0.5 round
down; if above 0.5 round up. If it is precisely 0.5 then rounding down is as
valid as rounding up. If it's absolutely necessary that something should
always be rounded up there's a function for it, ditto with rounding down.
0 Kudos
Message 3 of 4
(2,843 Views)
Actually, for rounding to integer, LabVIEW uses a statistically unbiased rounding method to the nearest even value. The number 0.5 is rounded to 0.0 as 1.5 is rounded to 2.0. When coercing to a fractionnal increment, maybe there is something similar involved, but it is harder to check...


LabVIEW, C'est LabVIEW

0 Kudos
Message 4 of 4
(2,843 Views)