File Size After WDF to TDMS Conversion

[steveh01]

I am using the "Convert to TDM or TDMS.vi" to convert a WDF file to a TDMS file, and for some reason the TDMS file size becomes nearly four times what the WDF file was.  The original file has 17,757,214 data values on a single channel.  For 16-bit (2-byte) data, that should be 2*17,757,214 Bytes ~ 35,514 KB.  Indeed, the original file size says it is 35,317 KB in the folder.

After conversion, the new TDMS file size is 138,730 KB in the folder, or 3.93 times what it was.  What is causing this?  I have read the article about why TDMS files are sometimes larger than they should be, and I have tried using the TDMS Defragment function, but with no effect.  There is no explicit time channel associated with the data.  I have tried modifying the data chunk size input, with no effect.  The only thing I can figure is that "Convert To TDM or TDMS.vi" must be storing data values using a full 64-bit format (~4 times the 16-bit data resolution).  Is this true?  Can this be changed using some input, or by reading and converting the WDF file with some other function?

[crossrulz]

Was the data stored as 16-bit ADC data or as scaled data in the WDF file?  My guess is that is was scaled data, which would be interpreted at a DBL.  A DBL is 64 bits in length.

[steveh01]

It was acquired using a Yokogawa DL850E Scopecorder with a 10:1 probe, so it has to have some scaling factor in the process.  However, I don't think that would make much of a difference once it reaches the file.  The WDF file size indicates that it is 16-bit data.  (I can also translate it into a WVF of comparable size using the WDF to WVF converter.)  Do you know how I can definitively tell whether it is 16-bit ADC data or scaled data in the WDF file?  I can see when I open up the TDMS file with the MS Excel Importer, that the values are "DT_DOUBLE" type.  Is there a way to tell the "Convert to TDM or TDMS.vi" what type of data to use when it writes the TDMS file?

[K C]

Hi Steve,

Few years back I made a WDF converter to JPG and CSV files. I used a WDF plug-in. You can extract the data from the WDF file with all other information present and stored it in another format (like CSV)  In my folder I see a WDF file size of 1.5M being reduce to a 48k CSV file.

Maybe this is an option. 

Kees

[steveh01]

K C,  I am using the Yokogawa_WDF plugin (http://www.ni.com/example/31611/en/) already.  That's what the "Convert to TDM or TDMS.vi" uses in LabVIEW to translate the data.  I would think that if you went from 1.5M WDF file down to a 48k CSV file, then you lost data.  Your conversion sounds like the MS Excel TDMS add-in, that only allows you to load as many data points as Excel can handle.  Only, in your case, it may be only as many data points as you put into your picture.  In my case data loss is unacceptable.

It sounds to me like there isn't an established way to keep the file size approximately constant using the WDF plugin and Convert to TDM or TDMS.vi.  So for now, I will either have to live with 4x the file size, or maybe do a second translation from one TDMS file to another.

It also looks like the Yokogawa_WDF plugin and/or Convert to TDM or TDMS.vi need an update.  Not only to check the WDF data size and write the same data size to the file, but also to handle multiple blocks of data from the WDF file. (See customer review from http://www.ni.com/example/31611/en/.)

[K C]

You can extract the data and properties of each channel / signal from the WDF file and store it in binary files if won't get any smaller than that without compression. That is what I did and used it to create pictures and CSV files. Without conversion to TMDS

Kees

[steveh01]

Do you have an example VI or a picture and description of how to do that?  Do I use the Yokogawa_WDF plugin with some other LabVIEW function?

[K C]

I used the Yokogawa plug-in. Here you have my project with the WDF to JPG converter.

The VI 'Proc WDF file to Data' extracts the data

Kees

[steveh01]

Thanks K C.  Reviewing your code did help me control the TDMS file size.  I had to install the OpenG libraries first to get it to run.  But now at least I can change the values from 64-bit doubles down to 32-bit single precision floating point numbers.  That cuts the file size in half.  The precision is still overkill for 16-bit ADC data, but better than before.

I can also see what caused your original WDF to CSV file conversion to drop your file size so dramatically.  You had a file with 139 blocks of data (as can be seen in the NumBlocks property), and the conversion only read the first block.  I am attaching some code to illustrate.  I used your example file, and one of my own, and converted each of the files using the standard "Convert to TDM or TDMS.vi" (with Yokogawa_WDF data plugin), and a VI that I wrote.  The resulting file sizes match these calculations (You'll have to run the conversion function to produce the TDMS files for my file, as they were too big to upload):

Method 1 (M1): Convert to TDM or TDMS.vi
1544K  * (1 / 139 blocks) * (64 bit double / 16 bit ADC) = ~44K.  (See "Example File 4Sig_139BlocksM1.tdms".)
35,317K * (1 / 1 blocks) * (64 bit double / 16 bit ADC) = ~140,000K.   (See "My File 1Sig_1BlockM1.tdms".)

Method 2 (M2): WDF_to_TDMS.vi
1544K  * (1 / 139 blocks) * (32 bit float / 16 bit ADC) = ~22K.  (See "Example File 4Sig_139BlocksM2.tdms".)
35,317K * (1 / 1 blocks) * (32 bit float / 16 bit ADC) = ~70,000K  (See "My File 1Sig_1BlockM2.tdms".)

So, as you can see, the number of data blocks makes a big difference in the end file size.  And if you have multiple trigger events that you want to capture, then you won't get it all converted over to TDMS.

I hope this is helpful to others.  However, there is currently a caveat on using my code.  For some reason some timing information is getting cut off when my WDF_to_TDMS.vi is used.  (With Method 2 selected, the start time for the example file is 10:56:11.000 AM rather than the 10:56:11.230 AM displayed when viewing the WDF file or the TDMS file converted using method 1.)  Hopefully I can correct this soon.

So, the main issue now is finding out how to access all of the data blocks within the WDF file.  Does anyone know how?

[steveh01]

This attachment has the time code problem fixed.  Now we just need someone who knows how to access multiple data blocks in the WDF file.