LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Difference in data conversion when provided as a single integer or in an array

Solved!
Go to solution

I was trying to do some statistical analysis in python and wanted to implement it in LabVIEW. My results didn't match even with the same logic applied. This is when I found out that there is a difference in data conversion when provided as a single integer or in an array. I am providing double data and converting it to a signed 8-bit integer. See the conversion of 5.5, 9.5 as for a few examples. The array output is 5 but provided as a single integer for conversion it gives 6. Similar is the case for 9.5 and such other numbers. I have attached a test VI and also a SS of what I'm getting as output. The VI generates numbers 0 to 21 with a step of 0.1. I don't know what am I doing wrong or is there a bug in my LabVIEW. Please help.

0 Kudos
Message 1 of 9
(241 Views)

This might be of some explanation

 

But you are right... Interesting....

-------------------------------------------------------------------
Unfortunately, most readers of this Forum, including some real Experts, have not mastered the skill of being able to read the code that Posters fail to post. If we cannot see "what you did wrong", we are unable to tell you how to fix it. (Bob Schor 28 August 2018)
0 Kudos
Message 2 of 9
(226 Views)

Hi yp,

 

an additional explanation is found in the definition of IEEE754 float format.

 

Adding 0.1 repeatedly will not give expected results as that value cannot be represented by float values precisely!

Best regards,
GerdW
CLAD expired, using LV2011SP1 + LV2017 (+LV2019 sometimes) on Win7+cRIO
Kudos are welcome Smiley Wink

Message 3 of 9
(216 Views)

Yes, Sir, I understand how rounding works but my question is why is it working differently when provided in an array than done individually?

0 Kudos
Message 4 of 9
(214 Views)

Hi yp,

 

adding 0.1 ten times will (most probably) not result in 1.0!

Best regards,
GerdW
CLAD expired, using LV2011SP1 + LV2017 (+LV2019 sometimes) on Win7+cRIO
Kudos are welcome Smiley Wink

0 Kudos
Message 5 of 9
(196 Views)
Solution
Accepted by yp96

GerdW is right, generate your list, then right click and change your formatting to 20 significant digits instead of 6. Now it makes more sense. It's not the fact that it's an array, it's the fact that the number in the array isn't exactly 5.5- it's 5.49999999999999644.

 

If you make an array and manually fill it with values of 5.5 and 9.5, they work as it does with the single number.

 

sigfigs.png

 

 

 

0 Kudos
Message 6 of 9
(167 Views)
Highlighted

@yp96 wrote:

Yes, Sir, I understand how rounding works but my question is why is it working differently when provided in an array than done individually?


Set your display format on the double array to 20 digits of precision and you'll see that 5.5 is not actually 5.50000000000....... but 5.4999999...644.  It is less than 5.5 so it will round down.

0 Kudos
Message 7 of 9
(166 Views)

@GerdW wrote:

adding 0.1 ten times will (most probably) not result in 1.0!


Just show more digits of precision and it becomes clear...


GCentral
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
0 Kudos
Message 8 of 9
(165 Views)

@yp96 wrote:

Yes, Sir, I understand how rounding works but my question is why is it working differently when provided in an array than done individually?


The actual number is not rounded just because you're viewing it as rounded. If you will index out the number from the array you will get the same result. The generated 9.5 in the array does not equal the 9.5 that you typed into the control manually.

0 Kudos
Message 9 of 9
(147 Views)