LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Inexplicable Rounding

I'm attempting to use the Scan from String funtion in order to change a string into a double precisions float, but I keep on getting a rounded result despite the fact that my input string is only 15 digits. Labview is rounding it to only one significant digit. I'm wondering if there's a way to fix this easily. Attached are pictures of the VI in use and the results of probes attached to the input and output

0 Kudos
Message 1 of 11
(1,298 Views)

What does your Measurment indicator say?

 

Are you talking about what probe 8 is showing?  That is actually 5 significant digits, not 1.  Probes have their own display format and probably round to fewer digits.  It doesn't mean the value on the wire is rounded to something different.

 

You could try creating a custom probe for double precision and set the display format of that to be 15 significant digits and place that custom probe on the wire.

0 Kudos
Message 2 of 11
(1,273 Views)

I'd say just forget about what the probe is saying and make sure that what is on the indicator is correct.

Bill
CLD
(Mid-Level minion.)
My support system ensures that I don't look totally incompetent.
Proud to say that I've progressed beyond knowing just enough to be dangerous. I now know enough to know that I have no clue about anything at all.
Humble author of the CLAD Nugget.
0 Kudos
Message 3 of 11
(1,241 Views)

You attached a picture of your VI, rather than the VI itself.  I can't execute or test a picture, so I tried to re-create the basic code.

Dbl Conversion.png

If you run this (drag this LabVIEW Snippet into a LabVIEW 2016 Block Diagram, where it will become LabVIEW code), you'll see that Resulting String is 9999999.949137.  Sounds like this works just fine, to me.

 

Bob Schor

 

P.S. -- if you don't have LabVIEW 2016 (I can't tell your LabVIEW version because (a) you didn't tell us, and (b) you didn't include your VI), you'll just have to trust me.

0 Kudos
Message 4 of 11
(1,178 Views)
@Bob_Schor wrote:

You attached a picture of your VI, rather than the VI itself.  I can't execute or test a picture, so I tried to re-create the basic code.

Dbl Conversion.png

 

 Bob_Schor, but brendanevers is using Scan from String, not Scan to String - I'm afraid that your example is not relevant...

 

@brendanevers wrote:

I'm attempting to use the Scan from String funtion in order to change a string into a double precisions float, but I keep on getting a rounded result despite the fact that my input string is only 15 digits. Labview is rounding it to only one significant digit. I'm wondering if there's a way to fix this easily. Attached are pictures of the VI in use and the results of probes attached to the input and output

 brendanevers, what are the settings of your numeric indicator? You should then setup it to display 15 significant digits, most probably you have now just 5-6.

 

logos_middle.jpg

0 Kudos
Message 5 of 11
(1,054 Views)

Oops, you are absolutely right.  If it had been a VI that was attached (instead of a picture), I would have noticed (when I ran the code) my stupid error.  But it is easily fixed:

Scan from String.png

Note that if you don't "adjust" the Display Format of the Double Out to 6 digits of Precision (or set it with a Property Node, as I do here), it will show 1E+7, the correct result to 8 or so significant digits.  So, again, without having the actual code to inspect, I have no idea why the Original Poster got the Result he (or she) wasn't expecting.

 

Bob Schor

0 Kudos
Message 6 of 11
(1,038 Views)

@Bob_Schor wrote:

Oops, you are absolutely right.  If it had been a VI that was attached (instead of a picture), I would have noticed (when I ran the code) my stupid error.  But it is easily fixed:

Scan from String.png

Note that if you don't "adjust" the Display Format of the Double Out to 6 digits of Precision (or set it with a Property Node, as I do here), it will show 1E+7, the correct result to 8 or so significant digits.  So, again, without having the actual code to inspect, I have no idea why the Original Poster got the Result he (or she) wasn't expecting.

 

Bob Schor


While you're at it, can you help me debug this xml code?  Here's a snippet of the code I am trying to fix.

not helpful.PNG

Bill
CLD
(Mid-Level minion.)
My support system ensures that I don't look totally incompetent.
Proud to say that I've progressed beyond knowing just enough to be dangerous. I now know enough to know that I have no clue about anything at all.
Humble author of the CLAD Nugget.
0 Kudos
Message 7 of 11
(1,027 Views)

Bill,

     Is your message intended for me (I posted a LabVIEW Snippet, which is (a) immediately "visible" in the Post, and (b) easily converted to LabVIEW code by dragging into a LabVIEW Block Diagram of the stated Version or later) or for the Original Poster (to illustrate the relative difficulty of dealing to PNGs "out of context")?  I'm happy to be shown the "Error of My Ways" ...

 

Bob Schor

0 Kudos
Message 8 of 11
(1,020 Views)

Bill is likely beng sarcastic and poking fun at those asking for help debugging code they aren't willing to share so you can help.

0 Kudos
Message 9 of 11
(1,016 Views)

I thought that might be the case, but sometimes I come down perhaps a little too hard, myself, on people who do this ...

 

BS

0 Kudos
Message 10 of 11
(1,013 Views)