Turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page

03-24-2020 12:06 PM

Options

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report to a Moderator

I am trying to convert a 64 bit number to hex, and the normal TypeCast does not seem to be working.

0.000063092

Using TypeCast to a hex U8 array or U64 , yields a hex value of 0x3F108A0840000000

Using multiple websites, if I convert the number to hex, I get 0x3F108A084E7A83CC

It appears that the top 4 bytes are not being converted in LabVIEW for some reason.

Kenny

Solved! Go to Solution.

Solution

Accepted by topic author Kenny_K

03-24-2020 12:18 PM - edited 03-24-2020 12:25 PM

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report to a Moderator

Your value is actually 0.0000630919967079535127

Once you show more digits on your input and enter the correct value, the results agree.

(Note that the closest you can enter is 0.0000630919999999999984).

That series of zeroes at your end tells me that your DBL was probably derived from a lower precision representation number. Where does it come from?

03-24-2020 12:18 PM

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report to a Moderator

Change the display format of your constant from 6 significant digits to something higher like 18 digits of precision.

You'll see your constant is actually something else and the rounding to 6 sig. figs. made it look like 0.000063092

03-24-2020 12:32 PM - edited 03-24-2020 12:33 PM

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report to a Moderator

@altenbach wrote:

That series of zeroes at your end tells me that your DBL was probably derived from a lower precision representation number. Where does it come from?

Yes, as I suspected, your number is actually derived from a SGL, and converting it to DBL does not really give you more significant bits. Here's the proof!

Your original incorrect result is at the bottom.

03-24-2020 02:31 PM

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report to a Moderator

Interesting. I didnt realize that If I originally typed the value in a SGL, and then converted the numeric constant to a double, it would still retain the 0 afterwards.

I am trying to program the value 6.30920E-05 into a device. To write the number to the device, it needs to be converted to HEX.

Got it now, after I deleted the NC and started with a double first, before I entered the value manually.

Kenny

03-25-2020 05:46 AM

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Email to a Friend
- Report to a Moderator

Usually the SGLs 7 digit precision is plenty (you only use 6 in your post), but when looking at bit level things like this appears. 🙂

/Y