LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

trim to two decimal from double datatype

Hello,

 

I am trying to trim the decimal places in a number. ( say 1.2345 to 1.23). I have tried the method of number to fractional string and frac/Exp string to number. Since I am working on RT timing needs more precision digits(comparison is involved for time). Increasing the digits to 17 does not result in desired value. attached VI

 

Thanks,

Bharath

Thanks & Regards,
Bharath Kumar
GCentral
0 Kudos
Message 1 of 8
(4,523 Views)

The "precision" input specifies the number of decimal places to the right of the decimal point.

You could use "Number to exponential String", but converting it back to a double will result in conversion errors that leave non-zero values in the low order bits.  Maybe you just want to convert to s Single, or perhaps a Fixed Point number.

 

 

"If you weren't supposed to push it, it wouldn't be a button."
0 Kudos
Message 2 of 8
(4,519 Views)

Hi Bharat,

 

Since I am working on RT timing needs more precision digits(comparison is involved for time).

So you need more precision by reducing the number of relevant digits in the value???

What about using integer values like number of microseconds for your RT timing problem?

(Or integer number of ticks in a FPGA target?)

 

Increasing the digits to 17 does not result in desired value. attached VI

This will NEVER change with float values!

 Floats have a limited resolution and CANNOT store each value correctly.

When showing 17+ digits you will always notice rounding errors of floats!

Best regards,
GerdW


using LV2016/2019/2021 on Win10/11+cRIO, TestStand2016/2019
Message 3 of 8
(4,516 Views)

If you want to compare two FP numbers to a certain precision, then you need to multiply each number by 10 to the power of how many decimal places and then round the numbers.  Compare those two numbers instead.  (It even works for comparing to places left of the decimal point - just use a negative exponent.)


Edit:  I must've taken too many cold capsules at the same time.  It seems my response is only peripherally related to the subject.  Sorry!

Bill
CLD
(Mid-Level minion.)
My support system ensures that I don't look totally incompetent.
Proud to say that I've progressed beyond knowing just enough to be dangerous. I now know enough to know that I have no clue about anything at all.
Humble author of the CLAD Nugget.
0 Kudos
Message 4 of 8
(4,478 Views)

Many exact decimal fractions (e.g. 0.1) cannot be represented in binary unless you have an infinite number of bits ... and you don't!!! Comparing floating point numerics is always very dangerous. All computations are done in binary. "Decimal" is just a convenient cosmetic representation of the internal data that has limitations. Since it is only cosmetic, simply change the display format of the array elements accordingly.

 

In your VI, you have two lossy conversions. First you format the DBL into a string rounded to the nearest value with two decimal digits. The you scan it and it will get rounded to the nearest possible binary DBL value.

 

You could do all computations on integers with values 100x higher and scale it back at the very end and for display. Or you could compare the formatted strings, which is computationally more expensive of course.

Message 5 of 8
(4,467 Views)

Your comment about the float to number conversion not limiting to 2 digits is actually incorrect.  Either probe the array of strings or set an indicator and see.  It is actually the conversion back to a float from the string in your number array indicator that displays your issue.  It has the understandable issue if one understands the IEEE-754 conventions for floats or doubles.

 

Seeing how the float to string is correct (with 2 digits of precision), compare strings and NOT the numbers.

Help the Community (and future reviewers) by marking posts as follows:
If it helped - KUDOS
If it answers the issue - SOLUTION
0 Kudos
Message 6 of 8
(4,466 Views)

Hello,

 

I tried the NI solution to remove the digits after decimal. this was the method 1 suggested. I guess as the significant digits after decimal increases, it is safer to compare after converting to integers. And back to double for display purpose.

 

Thanks & Regards,
Bharath Kumar
GCentral
0 Kudos
Message 7 of 8
(4,430 Views)

Hi Bharath,

 

it is safer to compare after converting to integers.

It's not just "safer", it's the only safe way…

Best regards,
GerdW


using LV2016/2019/2021 on Win10/11+cRIO, TestStand2016/2019
Message 8 of 8
(4,428 Views)