Multifunction DAQ

cancel
Showing results for 
Search instead for 
Did you mean: 

define precision & compare

Hi all,
 
I have to compare a value say 5.5002, is within limits of 4.25 & 5.50. I am using range & coerce function (including upper & lower limit) to do that. This 5.5002, I got by interpolation. After formatting 2 digits of precision, I am comparing this 5.5002 as 5.50, but is not bound within limits. How can I do truncation of this single precision no. to 2 digits of precision & declare it well within limits?
 
Thanks for your help.
 
 
Regards,
 
Kousy
 
 
0 Kudos
Message 1 of 5
(3,334 Views)

I used to do a pretty lengthy prodedure to do this.

First I would convert the numeric value to a fractional string ( using convert to fractional and exponential string  function) with  precision set as 2

And then, convert it back to numeric ( using convert fractiona exponential string to number function) an get a floating integer value truncated to 2 digit precision.

Not a elegant method, but it worked for me Smiley Wink

0 Kudos
Message 2 of 5
(3,330 Views)

Similar idea if you're sure your numbers won't overflow:  multiply by 1000, round to int, compare with a range that is similarly scaled by 1000 & rounded.

-Kevin P.

ALERT! LabVIEW's subscription-only policy came to an end (finally!). Unfortunately, pricing favors the captured and committed over new adopters -- so tread carefully.
0 Kudos
Message 3 of 5
(3,326 Views)

Thanks Devchander. It worked fine. Its a roundabout way, still fine. Thansk once again.

Kousy

 

0 Kudos
Message 4 of 5
(3,280 Views)
Glad to have been of assistance
0 Kudos
Message 5 of 5
(3,276 Views)