LabVIEW

[Xu]

SGL with temperature degC or degF do not correctly display values -9, -8, ... -1, 0, 1, ..., 9. For example, when input 1, it displays 0.999994; input 2, it displays 1.99999 ....
Attatched is a test VI (LabVIEW 7.0 and 7.1).
Could anybody give an explanation?.
Thanks,

[altenbach]

Right-click on each, --> "format and precision" and select
- floating point
- 2 digits of precision (or whatever the quality of your data is, i doubt it is microdegrees)
- unselect "hide trailing zeroes" if desired.

(A SGL number is rather inaccurate, and e.g. "1" cannot be represented exactly in binary with the number of bits used for the mantissa.)

[JeanPierre]

Quote of the year:
"1" cannot be represented exactly in binary...

That's too funny altenbach! How many bits of precision do you need to represent "1" exactly in binary??
¦¬))))

[altenbach]

Sorry, this did not come out quite right! Thanks J-P.

.... Zero is even harder. :-)

You've probably seen the T-shirt: "There are 10 kinds of people: those who understand binary and those who don't" or similar...

------------ back to the problem ---------------------------------

Anyway, the problem here is actually with the associated units, which are internally stored as Kelvins and only converted to degC for display.

Set it to Kelvin and small enough integers will be accurate. In your case, 1 degC is actually represented as a fractional number (274.15) internally and does not have an exact binary representation.

[Xu]

Thank you all for reply.
You are right, 1 degC is represented as 274.15 internally.

BUT how the internal number is converted to degC?
It shows that 274.150 (SGL) - 273.150 (SGL) could have an exact binary representation 1 (with exponent = 0 and mantissa = 1). For examples:

274.150 (SGL) - 273.150 (SGL) results 1,
exponent = 0
mantissa = 1

275.150 (SGL) - 273.150 (SGL) results 2,
exponent = 1
mantissa = 1

276.150 (SGL) - 273.150 (SGL) results 3,
exponent = 1
mantissa = 1.5

......

[JeanPierre]

My guess is that display routines (that display a number in an indicator) are DBL regardless of the type of the control.
For 0 degC(SGL) the displayed value -6,10352E-6 is
the result of 273.15(DBL) - 273.15(SGL)

The internal value 273.15(SGL) is probably promoted to DBL with single precision (zero padding in the mantissa) and the is substracted 273.15(DBL) for display, resulting -6,10352E-6.

[Xu]

Thanks a lot, Jean-Pierre.
From what you posted above, I am quite sure that it is a defect of the conversion routine for the temperature unit. The reference temperature for degC (273.15) and/or degF is stored internally as a DBL.
273.15(SGL) - 273.15(DBL) = -6.10352E-6;
274.15(SGL) - 273.15(DBL) = 0.999994;
......
To solve this problem, there should be two reference temperatures, 273.15(DBL) and 273.15(SGL), for DBL and SGL with temperature unit respectively.