11-07-2019 09:33 AM - edited 11-07-2019 09:50 AM
I was trying to do some statistical analysis in python and wanted to implement it in LabVIEW. My results didn't match even with the same logic applied. This is when I found out that there is a difference in data conversion when provided as a single integer or in an array. I am providing double data and converting it to a signed 8-bit integer. See the conversion of 5.5, 9.5 as for a few examples. The array output is 5 but provided as a single integer for conversion it gives 6. Similar is the case for 9.5 and such other numbers. I have attached a test VI and also a SS of what I'm getting as output. The VI generates numbers 0 to 21 with a step of 0.1. I don't know what am I doing wrong or is there a bug in my LabVIEW. Please help.
Solved! Go to Solution.
11-07-2019 09:44 AM - edited 11-07-2019 09:49 AM
This might be of some explanation
But you are right... Interesting....
11-07-2019 09:48 AM
11-07-2019 09:50 AM
Yes, Sir, I understand how rounding works but my question is why is it working differently when provided in an array than done individually?
11-07-2019 10:07 AM
11-07-2019 11:06 AM
GerdW is right, generate your list, then right click and change your formatting to 20 significant digits instead of 6. Now it makes more sense. It's not the fact that it's an array, it's the fact that the number in the array isn't exactly 5.5- it's 5.49999999999999644.
If you make an array and manually fill it with values of 5.5 and 9.5, they work as it does with the single number.
11-07-2019 11:06 AM
@yp96 wrote:
Yes, Sir, I understand how rounding works but my question is why is it working differently when provided in an array than done individually?
Set your display format on the double array to 20 digits of precision and you'll see that 5.5 is not actually 5.50000000000....... but 5.4999999...644. It is less than 5.5 so it will round down.
11-07-2019 11:06 AM - edited 11-07-2019 11:07 AM
@GerdW wrote:
adding 0.1 ten times will (most probably) not result in 1.0!
Just show more digits of precision and it becomes clear...
11-07-2019 12:08 PM
@yp96 wrote:
Yes, Sir, I understand how rounding works but my question is why is it working differently when provided in an array than done individually?
The actual number is not rounded just because you're viewing it as rounded. If you will index out the number from the array you will get the same result. The generated 9.5 in the array does not equal the 9.5 that you typed into the control manually.