LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

decimal

Solved!
Go to solution

The output from this little VI gives 1.
Should have been 1.000000212346. Why?snip.png

 

 

 

0 Kudos
Message 1 of 2
(2,462 Views)
Solution
Accepted by topic author Michael.Koppelgaard

The value IS actually 1.000000212346 but you are only displaying 6 decimals. Right-click on your control and select Display Format and change Significant Digits from 6 to 13.

 

You could also have verified the value by subtracting 1.0 from it and see that the result is 2.12346E-7.

0 Kudos
Message 2 of 2
(2,439 Views)