From Friday, April 19th (11:00 PM CDT) through Saturday, April 20th (2:00 PM CDT), 2024, ni.com will undergo system upgrades that may result in temporary service interruption.

We appreciate your patience as we improve our online experience.

DIAdem

cancel
Showing results for 
Search instead for 
Did you mean: 

Mod function in Calculator

Hi everyone,
 
in DIAdem script,
 
Call Formulacalc("R3 := 0.075671 mod 0.0000001") gives 0
Call Formulacalc("R4 := 0.075672 mod 0.0000001") gives 1.164.....E-17
 
what could be the reason, does it have anything to do with 16 digit precision of floating numbers?
should i simply assume the values below E-16 as zero?
 
thanks in advance
 
Kaan
 
0 Kudos
Message 1 of 2
(4,250 Views)
Hi,

This is a rounding problem, depending on the precision of the float-values.

You can't assume every value below 1E-16 to be zero, because this kind of  floatingpoint problems occur typically 14-16 digits behind the first significant digit of your values. If you do calculations with larger numbers, the problem will show up above 1E-16.

It is hard to tell exactly how large the rounding errors are going to be on complex calculations. There is always the danger of getting much larger errors if very large and very small values are involved in one calculation.

Example:
1234567890.1-1234567890 gives 0.0999999046325684

The roundig error is already dangerously close to the first significant digit of the result. If you do additional calculations with the result, the error can get large enough to produce completely wrong results. In most cases, this is not an issue, but as always, it is a good idea to take care of possible errors.


0 Kudos
Message 2 of 2
(4,228 Views)