floating point precision errors

[rbkup]

Hello - I have dug around a bit on this site concerning loss of precision for floating point numbers, and I have seen articles blaming this on the architecture, and one claiming that the problem was fixed in CVI 4.0. I am using LabWindows version 7.0, and I still see this problem. I have a double precision variable initialized to the value 16.62. When I start debugging, it shows up with the value 16.6200008. This requires a high-precision calculation, and it throws off all results following. I have tried this in two other environments, MS Visual C++ and Paradigm C++, and I do not get the same result, so I can't see this as an architectural issue. Any other suggestions?

[bilalD]

Hello

I initialized a double to 16.62, and in the debug tooltip, the value that got assigned was 16.620000000000001, which was the same for VC++ ( I dont have Paradigm C++ ). I tried this in CVI 7 and 7.1.
Did I miss anything?

Bilal Durrani
NI

[rbkup]

After digging around a bit more, I figured out that the value was being converted to a float when it was passed into a function, which is where the 16.6200008 came in. After I spoke to a support engineer at NI, I think what is really missing is a 'loss of precision' warning and maybe a problem with the number of significant figures that are displayed I think. VC++ will only display 4 sigfigs for a float & 8(?) for a double, which seems more in line with what the architecture supports. So it did turn out to be my error, but it would have been helpful to have the compiler warning about the type demotion... For a tool that is frequently used for scientific calculations, where errors at the 6th decimal place are sometimes very problematic, it would really be a 'ni
ce to have' warning.

[bilalD]

I've created a suggestion for this warning in CVI.

Thanks

Bilal