LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Difference in data conversion when provided as a single integer or in an array

Solved!
Go to solution

I was trying to do some statistical analysis in python and wanted to implement it in LabVIEW. My results didn't match even with the same logic applied. This is when I found out that there is a difference in data conversion when provided as a single integer or in an array. I am providing double data and converting it to a signed 8-bit integer. See the conversion of 5.5, 9.5 as for a few examples. The array output is 5 but provided as a single integer for conversion it gives 6. Similar is the case for 9.5 and such other numbers. I have attached a test VI and also a SS of what I'm getting as output. The VI generates numbers 0 to 21 with a step of 0.1. I don't know what am I doing wrong or is there a bug in my LabVIEW. Please help.

0 Kudos
Message 1 of 9
(2,308 Views)

This might be of some explanation

 

But you are right... Interesting....

========================
=== Engineer Ambiguously ===
========================
0 Kudos
Message 2 of 9
(2,293 Views)

Hi yp,

 

an additional explanation is found in the definition of IEEE754 float format.

 

Adding 0.1 repeatedly will not give expected results as that value cannot be represented by float values precisely!

Best regards,
GerdW


using LV2016/2019/2021 on Win10/11+cRIO, TestStand2016/2019
Message 3 of 9
(2,283 Views)

Yes, Sir, I understand how rounding works but my question is why is it working differently when provided in an array than done individually?

0 Kudos
Message 4 of 9
(2,281 Views)

Hi yp,

 

adding 0.1 ten times will (most probably) not result in 1.0!

Best regards,
GerdW


using LV2016/2019/2021 on Win10/11+cRIO, TestStand2016/2019
0 Kudos
Message 5 of 9
(2,263 Views)
Solution
Accepted by yp96

GerdW is right, generate your list, then right click and change your formatting to 20 significant digits instead of 6. Now it makes more sense. It's not the fact that it's an array, it's the fact that the number in the array isn't exactly 5.5- it's 5.49999999999999644.

 

If you make an array and manually fill it with values of 5.5 and 9.5, they work as it does with the single number.

 

sigfigs.png

 

 

 

0 Kudos
Message 6 of 9
(2,234 Views)

@yp96 wrote:

Yes, Sir, I understand how rounding works but my question is why is it working differently when provided in an array than done individually?


Set your display format on the double array to 20 digits of precision and you'll see that 5.5 is not actually 5.50000000000....... but 5.4999999...644.  It is less than 5.5 so it will round down.

0 Kudos
Message 7 of 9
(2,233 Views)

@GerdW wrote:

adding 0.1 ten times will (most probably) not result in 1.0!


Just show more digits of precision and it becomes clear...


GCentral
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
"Not that we are sufficient in ourselves to claim anything as coming from us, but our sufficiency is from God" - 2 Corinthians 3:5
0 Kudos
Message 8 of 9
(2,232 Views)

@yp96 wrote:

Yes, Sir, I understand how rounding works but my question is why is it working differently when provided in an array than done individually?


The actual number is not rounded just because you're viewing it as rounded. If you will index out the number from the array you will get the same result. The generated 9.5 in the array does not equal the 9.5 that you typed into the control manually.

0 Kudos
Message 9 of 9
(2,214 Views)