number compare 0

[pwaszkewitz]

I have noticed something peculiar in TS 4.1.

When I have a small number, like 1e-14, the following comparisons hold in an expression:

• Locals.VerySmallNumber == 0 -> False
• Locals.VerySmallNumber == 0.0 -> False
• 0 == Locals.VerySmallNumber  -> True
• 0.0 == Locals.VerySmallNumber  -> True

I might have understood the (0 == ...) bit as some implicit C-style data type conversion as the statements in general have C operator syntax, but 0.0 in C would be a double literal and therefore should compare False at that precision level. But apparently it depends on sequence?

• Is that behavior by design?
• Is that true in all TestStand versions?
• Is there a way to enforce a comparison with double precision regardless of the sequence (This is perhaps a weird hangup, but we like to have the constant as the point of comparison in front of the operator to avoid the stupid "Variable = 0" instead of "Variable == 0" mistake)?

See the attached sequence file

Best regards,

Peter

[James_Grey · Accepted solution]

Floating point comparisons in TestStand are performed at approximately 14 decimal digits of precision.

However, there is a known inconsistency that applies only to comparisons between a number that is EXACTLY zero and a very small number that is approximately 1e-14 or less. In this case, whether TestStand considers the numbers to be equals depends on which side of the comparison the exact zero is on.

This will be fixed in a future version of TestStand.

[pwaszkewitz]

Thank you for the clarification.

Peter