Floating number conversion inside of LVDateTimeRec

[ViltBalint]

Hi Everyone,

Shortly my problem: I tried to convert a LV.TimeStamp to LVDateTimeRec and it seems to me, the conversion works on a different way as i expected. It means, if the "fractional second (FS)" part of data structure is bigger than a treshold, the value will be casted on a weird manner for me. For example (see attached picture): FS = 0.999999 --> OK, FS = 0.9999995 --> NOT_OK (converted to 1).

Could someone provide information for me what is happening exactly?

Balint

[altenbach · Accepted solution]

What is the display format for the indicator?

(What you see is not what the underlying data really is, but a formatted value based on the indicator properties. There is no "conversion")

[aputman · Accepted solution]

The default display format is 6 significant digits.  0.999999 meets that criteria, so the entire number is shown in the indicator.  0.9999995 has seven digits, so the value is rounded up (if using the default display format).

[ViltBalint]

Thanks for the fast ansewers, it is clear 🙂 Just an additional question; why are different the probed and highlighted value of num?

[aputman]

You can create a custom probe, if it's necessary.  Again, it's just the display of the data that is different, not the data itself.  

[ViltBalint]

Okay, I did not understand why I see "1,00" during highlighted execution. But if I understand well, in current case (maybe configurable(?)) highlighting works as an indicator with 3 digit precision. Right?

[altenbach]

I don't think the formatting for the floating numbers during execution highlighting is configurable. It shows with limited resolution for clarity.

[ViltBalint]

I have an additional question: if I have a floating number (0.999), could I create a formatting settings which results 0.99 in case of 3 significant digits instead of 1.00?

[altenbach]

---

@ViltBalint wrote:

I have an additional question: if I have a floating number (0.999), could I create a formatting settings which results 0.99 in case of 3 significant digits instead of 1.00?

---

You could use a plain string indicator and generate any kind of misleading display you want. For example you could format the number with many decimal digits and then truncate the string two digits after the decimal point before displaying it.

Alternatively, you could do the calculation numerically and turn your 0.999 into a 0.99 before sending it to the display. It's just math. (e.g. multiply by 100, round to -inf, divide by 100 or similar)