LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

How do i avoid rounding

Hello,

 

 

I am trying to subtract an array of very small numbers from 1. The resulting output ends up rounding up as an array of 1s. I am expecting to get numbers very close to 1 (say 0.9999999997 for exampe) but not exactly 1.  I have tried to ensure that both inputs to the numeric operator have the same type DBL/SGL etc. I have also tried scaling both inputs up and dividing after.

 

I know this must be easy but I dont have a ready solution  🙂  VI attached. I need both probes 64 and 65 to maintain a good number of precision digits. Thanks.

 

 

 

[BADGE NAME]

0 Kudos
Message 1 of 4
(3,915 Views)

As you didn't store your "very small numbers" as default values, the attached VI doesn't show much except the code itself.

However, your numeric control/indicator is still in default configuration for showing 6 digits of precision. You might want to increase that in the controls/indicators property dialog to show e.g. 15 digits of precision.

 

Norbert

Norbert
----------------------------------------------------------------------------------------------------
CEO: What exactly is stopping us from doing this?
Expert: Geometry
Marketing Manager: Just ignore it.
0 Kudos
Message 2 of 4
(3,905 Views)

Increase the digits of precision

Capture 2.PNG

 

to get

 

Capture.PNG

 

Ken

Message 3 of 4
(3,895 Views)
  • When attaching an example VI, make sure the controls contain typical default data so we can see what you are talking about.
  • Not the cause of your problem, but mind your representations. that "1" diagram constant should be DBL, not SGL. Notice the coercion dot?
  • DBL values have a precision of about 15 decimal digits. You seem to be worried about the displayed precision of controls and indicators.
  • The displayed precision is only cosmetic and you can change the display format to anything you want (right-click...display format"). The display format does not affect the underlying data, so if you would square the result and subtract  from 1 again, you would get the original data back (within the limits of DBL computations). Try it.
  • Your VI does not contain any "probes", but if you want a probe that shows values with higher precision, you can create custom probes.
  • You cannot change the format of execution highlighting.

 

0 Kudos
Message 4 of 4
(3,885 Views)