LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Rounding Off Issue in Double to String

Solved!
Go to solution
Solution
Accepted by topic author Karthik_Abiram

@Karthik_Abiram wrote:


LabVIEW should be better handling these things.

 

So, is it not possible to represent a number like 488.345(488.34500000000003) as 488.345 exactly?


I agree that the behavior you're seeing is a bug. 488.345 is not exactly representable in floating-point, but 488.375 is, and the truncation it exhibits is inconsistent with other numbers that round half to even.

Here's a very simple example involving exact storage values:
to 1 digit,  1.75 --> 1.8
to 2 digits, 1.375 --> 1.37

I'm not aware of any rounding mode that should behave that way (other than randomized!). I've filed CAR 515549 on this.

0 Kudos
Message 11 of 12
(1,051 Views)

@JLewis wrote:
I agree that the behavior you're seeing is a bug. 488.345 is not exactly representable in floating-point, but 488.375 is, and the truncation it exhibits is inconsistent with other numbers that round half to even.
Here's a very simple example involving exact storage values:
to 1 digit,  1.75 --> 1.8
to 2 digits, 1.375 --> 1.37

I'm not aware of any rounding mode that should behave that way (other than randomized!). I've filed CAR 515549 on this.

I agree, and thanks for filing the CAR!

------
Using LabVIEW since 2012. Certified LabVIEW & TestStand Architect
0 Kudos
Message 12 of 12
(1,039 Views)