different values beetween front page and saved data

[shadok]

Hello,

I have problems to saved data correctly. I acquire data with Labview 6.0 and a counter/timer card. These data appear on my front page. But when I save them in a table file, their format is modified, rounded.

For exemple: on my front page values are 20.675 , 21.012 and in my saved file these values are 20.750 and 21.000, even if I checked all the numbers format in the diagram, and declare them in extended precision.

Where is my mistake ?

Thank you for your help

[GerdW]

Hello shadok,

can you post some code? It would be much easier to find the misbehaviour...

[shadok]

here is my code.

Thanks

[Dennis_Knutson]

It doesn't matter what precision you specify for display or the numeric representation. When you use the Write to Spreadsheet function, you have a format specifier of %3f. This specifier writes a floating point number with a width of 3 and no decimals. The default is %.3f (note the decimal point) which says to write a floating point number with 3 significant digits.

[GerdW]

Hello shadok,

as Dennis wrote by now, you specify %3f as precision for writing. This means "floting point with 3 significant digits" (means NOT: "3 digits after decimal point"!!!).
Change this to %.3f. Or even better to %.4f, as all your data is U32 divided by 80 giving a resolution of 0.0125...

Hope this helps

[Joe_H]

The problem lies in the format that you are writing to the file, on the write from spreadsheet file .vi ther is a formatting that you set. right now you have it set to %.3f which means you will have three digits past the decimal in floating-point. you will need to change this. look at the help and follow the sytax to change it.

[shadok]

I'm sorry it doesn't work!

In my file I have now 4 numbers after the decimal point, but the real value is still rounded at the nearest multiple of 0.125

It gives me 21.875 for 21.862 on the front page

[GerdW]

Hello shadok,

that's strange 🙂

Try this simple example. If this works, then something with your data is wrong...

[GerdW]

Hello shadok,

one more guess:
Have you set any range properties of your indicators? I couldn't find any, when loading your vi, but perhaps they are vanished after loading into LV7.1...
If you set the increment to 0.125 in the data/range properties your behaviour could happen.