In Range and Coerce Up and Low limit inclusiveness

[MogaRaghu]

The attached VI is a simple comparator to find out if an actual value is within limits and I want to give a true verdict even when the Actual is equal to either the High or Low limt.

But It does not seem to work...

[Sam_Sharp]

Set the display format to show the maximum precision (i used 20) and you will understand what is happening:

The numbers are not actually in range because of a very slight rounding error. There's probably only 1 bit difference between the two numbers but it means they're not equal. In the same way, if you were to equals the two numbers together they would not be the same.

One solution would be to multiply your numbers by 10^(required precision), round to nearest and then divide by the same power of 10.

[MogaRaghu]

Yes I am aware of this issue - precision of background calculatiuons and display precision.

But my point is this : For both upper and lower set points, I manually entered the values 23.26 and 18.05 and thats it. There cannot be anything more to them even if the precision is stretched to any digits. And as to teh Actual value that also is incremented by 0.01 as I have set the property.( I can understand that there can be a variance between processed and displayed precsion if the value is the result of some calculation )

In suhc a case how did the additional figures appear after 18.05 in Low Set point and also in Actual which must be only 3 decimals precise ??

[Sam_Sharp]

It's a computer science / binary / floating point representation of numbers thing: A floating point number is  an 'approximation' of the value entered. LabVIEW has to represent the number in a fixed number of bits (32) so it represents it in the binary version of a scientific number (e.g. 3.13E-1) - the mantissa (value) and then the exponent (scale).

You can read more about it here: http://en.wikipedia.org/wiki/Floating_point

[MogaRaghu]

Ok I guess I understand - even though I physically enter the value and stop with the second decimal, the way LV stores it for processing brings in some approximation error.

This was evident when I tried the same example I sent you on a 32bit machine and it worked correctly some times. But on my 64 bit laptop, it never worked correctly even once.

If this is the case then how would the excercise of  multiplying thenumber by 10^X  and then dividing by 10^X and rounding off work ? Anyway finally LV will approximate it ??

[Sam_Sharp]

The point of multiplying, rounding and then dividing is to try and force the numbers to be represented in the same way to a given acceptable precision (so you're making the numbers the same if they are approximately the same to an acceptable level of precision) - essentially truncating the bits and then returning it back to it's original value (where it will then approximate the representation of both numbersin the same way).

Putting this in-between the inputs of the in-range and coerce and your incoming DBLs should give you the desired effect.