Less than or equal to broken?

[jhg]

Download the vi and check out the internals, as they will explain exactly what I'm trying to do.

Briefly, I am writing a deMUX for a signal that has digital components of 0.25V, 0.5V, 1V, 2V, and 4V. The max signal being 7.75V and the min being 0V. For example, if the input is 0.25V, the 5 bit digital output will be 00001; if the input is 0.50V, the digital output will be 00010; if the input is 0.75, then the digital output will be 00011; ... and if the input is 7.75, then the digital output will be 11111.

In order to make the system realistic I have added tolerances to the search. So, the tolerance I picked is 0.1V (which is well less than the required 1/2 of the smallest voltage increment of 0.25V, which is 0.125V). Thus, any of the 32 possible voltages +/- 0.1V should decode correctly according to the <= circuit I have set up toward the bottom of my while loop in the attached vi. It does work for most, but it does not work for 1.1V, 2.1V, and 3.1V... and possibly some others. For these listed numbers, the comparison acts as a less than, rather than a less than or equal to.

Let me know if you need more information.


Cheers,
Joe

[tbob]

You are experiencing rounding off error that commonly occurs when subtracting two floating point numbers.  You may see a subtraction result of 0.1, but it may really be 0.0999999984, or 0.100000000000134, or something like that.  This is because of  the way floating point numbers are stored digitally.  If you change your tolerance to 0.10001, you will see that 1.1 now works.  This is always a problem when working with floating point numbers.  Instead of trying to test for equality or greater or less than, it is best to test if a number is in range by so many decimal places.  Like instead of testing x=1, test for x>0.999999 and x<1.000001.  In your case, test for the subtraction to be less than 0.10001, don't use <=, just use <.

[jhg]

Thanks. I never realized floating point would do this to such nice numbers.

Joe

[johnsold]

Joe,

It is not so much floating point as binary (base 2) versus decimal (base 10). In decimal one tenth is expressed exactly as 0.1. In binary the same fraction is an infinitely repeating expression. Reciprocals of prime numbers such as 7 will be repeating "decimals" in any base except the prime number istelf (or integer multiples thereof). Actually the requirement that a fraction be represented as repeating in any base is that the base and the denominator be mutually prime.

Lynn

[tbob]

Very good explanation, "Mr. Lynn".  I knew the problem but just couldn't express it in words.  You get stars for this one.

[altenbach]

To solve your problem, all you need is to "quantize" the number to your smallest step (0.25 in your case) and convert it to an integer.

The attached code will probably produce the output you want. 😄

Message Edited by altenbach on 12-21-2005 09:28 AM