Showing results for 
Search instead for 
Did you mean: 

comparing problem

Hello together
This morning I wanted to compare some data values read from a txt- file (see attached vi). What I thought to be a one minute job, turned into a tricky work...
The job and the vi is simple, but doesn't work as expected. I just want to have the information whether each component in the array is equal to 0.2 or not.
But something seems to be strange with the data.
What's the problem of the data?
Thank you for any help
PS: I'm using LV8.0
0 Kudos
Message 1 of 4
Without actually being able to look at the data (I don't have LV 😎 I would hazard a guess that the problem is the rounding error of floating point values.

Since floating point values are only approximations for some values (0.2 being one of them) then some seemingly equivalent values may not be binary equivalent, thus resulting in a false "Equals" comparison.

Instead, check to see if the difference between the values is below a certain threshold (0.00001 for example).  This threshold simply has to be larger than the rounding error of the data type you are using, but smaller than any deviation you can tolerate in your comparison.

i.e. If abs(x-y) > Threshold

Look up your LV 8 help for the datatypes on your machine.

Hope this helps


Message Edited by shoneill on 04-20-2006 10:00 AM

Using LV 6.1 and 8.2.1 on W2k (SP4) and WXP (SP2)
Message 2 of 4

Hello shoneill

Thank you for your answer. You are completely right. This is a rounding error. I definitely should have known that, but obviously I have never come at this point before, so I was very confused. To see  wheter the value is equal to 0.2 I must check it in a certain range.

Thank you!


0 Kudos
Message 3 of 4
No problem, glad to be able to help,

I remember tearing my hair out with a similar problem where I was reading from a large text file at a specific index.  Certain data sets simply didn't load properly.

I spend a long time trying to find out why.  In the end, I had a SGL index for the file which, with certain index numbers, rounded to the "wrong" integer when reading the file.  This led to a 1-Byte shift when reading the file, and thus the data set was corrupted.

I really felt stupid after finding out the mistake, but it was a valuable lesson because I'm automatically wary of it ever since.

Moral of the story, maybe I should have let you worry longer before posting Smiley Tongue


Message Edited by shoneill on 04-20-2006 10:27 AM

Using LV 6.1 and 8.2.1 on W2k (SP4) and WXP (SP2)
Message 4 of 4