Too many digits of precision in imported data

[GS]

I store some data in a TDMS file that has four digits of precision. When I recall the data in DIAdem or import it into Excel there are many more digits there than I want. Do I just have to live with that or is there a way to limit the number of digits to just what I want?

George

[Brad_Turpin]

Hi George,

Are you saving the TDMS file with LabVIEW or DIAdem?  In LabVIEW, you wire a data array of a particular data type to be written to the TDMS file, so the precision used on disk will match that of the data type.  So if you're currently wiring a DBL type, converting that array to a SGL type prior to writing the array to TDMS will reduce the precision (and the disk footprint), though it may not reduce it enough for your taste.  If you are wiring waveforms to the TDMS file, I believe you are stuck with DBL precision.

If you are saving the TDMS file in DIAdem, and if you have DIAdem 10.1, then you have the option of using the new ChnQuantize() function to limit the resolution of the data both inside of DIAdem and when you save it to disk.  Unfortunately, DIAdem 10.1 has a bug for TDMS output when using custom quantization, though this will be fixed in the upcoming DIAdem 10.1 SP1.  Still, this doesn't necessarily reduce the number of displayed digits, it just reduces the accuracy so that similar values can be saved in a smaller binary footprint.  So this may not be what you're after.

Regardless of the resolution of the saved binary data, DIAdem loads all channel values as DBLs in the Data Portal, and it displays all the digits of precision corresponding to the DBL value it has in memory.  However, you can always change the displayed format of a channel value in a VIEW or REPORT table by changing that column's format string.  Try "d.ddd" or "d.ddde" for instance.

Ask if you have further questions,
Brad Turpin
DIAdem Product Support Engineer
National Instruments

[GS]

Brad,

I'm saving the data with LabVIEW so it sounds like there's not much I can do. The data is being imported into Excel with the TDMS importer. I told the guys who are working with the data that probably the best thing to do is limit the precision in Excel to the precision of the instrument (4 decimal places).

George

[Brad_Turpin]

Hi George,

Now that I know you're saving the data in LabVIEW and reading it in Excel, I have a much better idea what's going on.  If you show an array of these values on a front panel indicator, do they show the desired precision or the extra precision which you don't want?  If you're saving the data as a DBL array or probably even as a SGL array, then I would expect the precision to show up in Excel just as it looks in LabVIEW.  If you see too much precision in LabVIEW, then the simplest solution is to truncate the data in LabVIEW prior to saving it to file.  Of course you lose resolution by doing this, but it sounds like that extra resolution is either meaningless or more distracting than useful.

If on the other hand the data looks correct in LabVIEW but shows up with extra digits in Excel, I'd recommend that you send me the part of your LabVIEW code which is doing the writing.  I'm not sure why that would be the case.

By the way, are you using DIAdem 10.1 and the improved DataFinder these days?  Will it make sense for your groups to use the DataFinder Server Edition, or is the local DataFinder on each DIAdem client sufficient?  Thanks for all the Beta testing.

Brad Turpin
DIAdem Product Support Engineer
National Instruments

[GS]

I changed the front panel control to only show 4 digits of precision. The imported data still shows way more digits than that. I think this makes sense because I think changing the digits of precision of the control only changes what it shows. I pared my VI down to almost the bare minimum and attached it and simple VI that feeds data to the storage VI.

We're still using DIAdem 10.0. The wheels turn slowly here sometimes. However, I think it will make sense for us to upgrade and use the DataFinder server edition in the near future.

George

[Brad_Turpin]

Hi George,

This is what I am talking about-- you have to delete the precision that you don't want from the values in the LabVIEW array prior to saving those values to the file.

Brad Turpin
DIAdem Product Support Engineer
National Instruments

[GS]

I thought about doing that but it doesn't work (I tried your VI and imported the data to both Excel and DIAdem).  I think when you do the divide it converts the number back to a Single so nothing actually changes.

The only thing I that really works is to multiply by 1000, convert to an integer and tell the user that the number is scaled by 1000.

George

[Brad_Turpin]

Hi George,

OK, I sent that VI off to you a bit quickly yesterday.  You're right that when the number is scaled back to its original value as a SGL that the same "extra" precision issue crops back up.  I went ahead and populated the VI with default data and changed the "1000" constant to the data type DBL in order to force the values saved to TDMS file to also be of type DBL.  On my computer this now results in the 3 digits of precision I was aiming at-- in both DIAdem and Excel.

Let me know if you see something different on your end,
Brad Turpin
DIAdem Product Support Engineer
National Instruments

[GS]

Yep, that one works. Thanks much.

George