unit conversion causes error in results

[SunnyH]

Hello all,

Please help!  I found an annoying problem with unit conversions.  I'm running LabView 6.1.  In my program, I read in a data file containing calibration data, then my program adds units to the data.  The problem is: after adding units, the number would be a little bit off.  I've tried changing precision from SGL, DBL, EXT, and I still have the same problem. 

Attached is a VI showing the problem, together with screen shots of the diagram and front panel.


Thanks,

Sunny

[tbd]

Hi Sunny,

      Right-click on the "Input" control (Front Panel) choose "Format and Precision" and change the "digits of precision" from 5 to 8.  Input is being rounded-up from "0.89999962" to 0.90000.  If you do the same for "Before conversion" you'll see that all the values are really in agreement.!

Cheers.