Turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page

11-07-2019 09:33 AM - edited 11-07-2019 09:50 AM

Options

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report to a Moderator

Labels:

I was trying to do some statistical analysis in python and wanted to implement it in LabVIEW. My results didn't match even with the same logic applied. This is when I found out that there is a difference in data conversion when provided as a single integer or in an array. I am providing double data and converting it to a signed 8-bit integer. See the conversion of 5.5, 9.5 as for a few examples. The array output is 5 but provided as a single integer for conversion it gives 6. Similar is the case for 9.5 and such other numbers. I have attached a test VI and also a SS of what I'm getting as output. The VI generates numbers 0 to 21 with a step of 0.1. I don't know what am I doing wrong or is there a bug in my LabVIEW. Please help.

Solved! Go to Solution.

11-07-2019 09:44 AM - edited 11-07-2019 09:49 AM

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report to a Moderator

This might be of some explanation

But you are right... Interesting....

-------------------------------------------------------------------

Unfortunately, most readers of this Forum, including some real Experts, have not mastered the skill of being able to read the code that Posters fail to post. If we cannot see "what you did wrong", we are unable to tell you how to fix it. (Bob Schor 28 August 2018)

Unfortunately, most readers of this Forum, including some real Experts, have not mastered the skill of being able to read the code that Posters fail to post. If we cannot see "what you did wrong", we are unable to tell you how to fix it. (Bob Schor 28 August 2018)

11-07-2019 09:48 AM

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report to a Moderator

Hi yp,

an additional explanation is found in the definition of IEEE754 float format.

Adding 0.1 repeatedly will not give expected results as that value cannot be represented by float values precisely!

Best regards,

GerdW

CLAD expired, using LV2011SP1 + LV2017 (+LV2019 sometimes) on Win7+cRIO

Kudos are welcome

GerdW

CLAD expired, using LV2011SP1 + LV2017 (+LV2019 sometimes) on Win7+cRIO

Kudos are welcome

11-07-2019 09:50 AM

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report to a Moderator

11-07-2019 10:07 AM

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report to a Moderator

Hi yp,

adding 0.1 ten times will (most probably) not result in 1.0!

Best regards,

GerdW

CLAD expired, using LV2011SP1 + LV2017 (+LV2019 sometimes) on Win7+cRIO

Kudos are welcome

GerdW

CLAD expired, using LV2011SP1 + LV2017 (+LV2019 sometimes) on Win7+cRIO

Kudos are welcome

Solution

Accepted by yp96

11-07-2019 11:06 AM

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report to a Moderator

GerdW is right, generate your list, then right click and change your formatting to 20 significant digits instead of 6. Now it makes more sense. It's not the fact that it's an array, it's the fact that the number in the array isn't exactly 5.5- it's 5.49999999999999644.

If you make an array and manually fill it with values of 5.5 and 9.5, they work as it does with the single number.

Highlighted
Options

11-07-2019 11:06 AM

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report to a Moderator

@yp96 wrote:

Set your display format on the double array to 20 digits of precision and you'll see that 5.5 is not actually 5.50000000000....... but 5.4999999...644. It is less than 5.5 so it will round down.

11-07-2019 11:06 AM - edited 11-07-2019 11:07 AM

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report to a Moderator

@GerdW wrote:

adding 0.1 ten times will (most probably) not result in 1.0!

Just show more digits of precision and it becomes clear...

There are only two ways to tell somebody thanks: Kudos and Marked Solutions

Unofficial Forum Rules and Guidelines

11-07-2019 12:08 PM

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report to a Moderator

@yp96 wrote:

The actual number is not rounded just because you're viewing it as rounded. If you will index out the number from the array you will get the same result. The generated 9.5 in the array does not equal the 9.5 that you typed into the control manually.