In Range and Coerce

[rohrdt]

I am having some inconsistencies with the In range and coerce function.  I realize the problems with trying to representing numbers exactly in binary format.

In my main program I get one result, and when I created a test case in a different vi, I get the opposite result.

How can I account for this in my code?

Attached are pictures of both VIs.

Thanks

[altenbach]

You simply need to increase your tolerance to a level where the results are as desired. 😄

Message Edited by altenbach on 05-17-2006 12:34 PM

[rohrdt]

That's part of the problem, I can't.

We are testing whether a measurement is inside a specific interval to determine whether the measurement is in tolerance. 

So we can't widen that interval.

[altenbach]

You only need to widen it by a hair to take care of binary rounding. e.g. try to add 1e-12 to your tolerance. It won't make any real difference to the result.

[rohrdt]

Good idea.  I'll give it a try.

thanks for the suggestion

[rohrdt]

Nice suggestion, it worked great!!!

Thanks again