Home Community Discussion Forums Most Active Software Boards LabVIEW Topic

LabVIEW

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

digit precision error

digit precision error

‎03-07-2012 10:40 AM

Hi everyone...

 

can be considered a bug or is a my wrong setting?

 

Why when i write 0.4 in a control, the indicator give tom me 0.4000000000000000022 ???

 

Thaks to all

 

 

 

 

0 Kudos
Message 1 of 3
(2,175 Views)
Reply

Re: digit precision error

‎03-07-2012 10:45 AM

Im sure someone can explain it and i have a nagging idea about what is happening but cant put into words, if you change them from doubles to fixed point, it works fine

Please remember to accept any solutions and give kudos, Thanks


LV 8.6.1, LV2010,LV2011SP1, FPGA, Win7
0 Kudos
Message 2 of 3
(2,172 Views)
Reply

Re: digit precision error

‎03-07-2012 10:47 AM

Floating point error is normal

PaulG.

LabVIEW versions 5.0 - 2020

“All programmers are optimists”
― Frederick P. Brooks Jr.
Message 3 of 3
(2,168 Views)
Reply