LabVIEW

cancel
Showing results for 
Search instead for 
Did you mean: 

Rounding issues with Scan from string time conversion?

Solved!
Go to solution

I have the following input string:

2021.08.01 - 23.10.14 Dispense.tdms

 

I added the date format to the file name so that the files sort and save in a orderly fashion that is not modified by saving the file and other changes:

 

YYYY.MM.DD - HH.MM.SS

 

Now I am trying to convert the string back to a date and use them in Excel. For some reason it looks like the conversion from timestamp to dbl is getting rounded off or something. 

 

Converting the input string "2021.08.02 - 23.15.16 Dispense.tdms" to a LabVIEW timestamp yields the correct LabVIEW time. When the LabVIEW timestamp gets converted using DBL I get a number that looks like it is getting rounded off or something. 

flycast_0-1629397481540.png

 

Playing with the conversion a bit more it seems that if I convert midnight on 8/2/2021 then I get 3710730000. 

 

flycast_1-1629397775668.png

 

If I convert midnight on 8/3/2021 I get 3710810000.

 

flycast_2-1629397844744.png

 

So then if I convert 1 second after midnight on 8/3/2021 I get the same result as midnight - 3710810000.

flycast_3-1629397920395.png

 

Why?

 

0 Kudos
Message 1 of 2
(761 Views)
Solution
Accepted by topic author flycast

What you are seeing is LabVIEW rounding the number to fit the display properties you set or let default. (doubles default to 6 significant digits)

 

DefaultDefault10 significant digits10 significant digits

BTW: LabVIEW uses the full precision of the number internally for all mathematics, but rounds the display to match the format set. That's why your math comes out correct.

 

========================
=== Engineer Ambiguously ===
========================
Message 2 of 2
(758 Views)