09-28-2011 10:52 AM
Hi everyone,
I have two different data which I'm doing very simple calculation with them; substraction.
First value is 1,640220000000000e+18
Second value is -1,640217348851742e+18
Obviously, these cannot be estimated as equal, the difference between them is 2651148258304. But LabVIEW calculates it as zero.
Why this error happens?
Regards,
Ozgur
09-28-2011 11:13 AM - edited 09-28-2011 11:17 AM
Please post a VI that demonstrates this.
What version of LabVIEW are you using? What is the representation of your numbers (single, double, extended)?
Is the fact you are using a comma instead of a decimal point causing you a problem?
And the difference between them is actually about 3.28e18 not 2651148258304.
Also, remember that many floating point numbers can't be represented exactly due to the binary format that PC's use internally.
09-28-2011 11:50 AM
What unit are they in? Were they at least a I64?
09-28-2011 12:52 PM
I have LabVIEW 2010, and the values are obtained in a calculation inside MathScript.
So, i don't decide the data type. There will be a way, I don't know that either.
In MATLAB, double precision is enough to represent that number. I really don't get why this happen in LV.
I will send a sample to show you that, as soon as possible.
Regards,
ozgur
09-28-2011 12:58 PM
You might want to post your code and continue this discussion in the MathScript forum.
09-29-2011 01:28 AM
Here is a sample.
The sum of those are estimated zero in the t(i)=..../(T11(i)+(T12(i)/(rho*c))+(rho*c*T21(i))+T22(i)); the denominator.
I will post a massage in MathScript forum also.
Regards,
ozgur
09-30-2011 10:19 AM
The new thread for this topic on the MathScript forum can be found here.