TDM file larger than TDX

[RussellSenior]

I am running a batch script that downloads some data, analyzes it, and stores it for future reference, but the TDM file size is about 2.5MB, when the TDX files size is only 0.5MB.  It seems odd that my meta data is ~5x the size of my actual data.  

Any idea what could be going on here, and any suggestions to fix it? 

[Otmar]

Hello Russell,

The one reason that I can see why DIAdem would create a large TDM vs TDX file is if you added a lot of meta data to your file. The TDM file is in XML format, the TDX Is a much more efficient binary file. If you have a lot of meta data on the file/group/channel levels of your data file, the TDM file could potentially become pretty large. XML is not the most efficient way to store data, but the advantage is that you can use most XML editors to look into the file. It may be worth checking is somehow a lot of meta data was added to the file.

You can always send me a TDM/TDX file and I will be happy to check on our end to see if there is anything fishy going on with the file ...

Let me know how you'd like to proceed,

     Otmar

[AndreasK]

String channels are also stored in the TDM file itself.

So I see three possible reasons:

• A lot of channels with only a few values
• A lot of properties attached to groups/channels
• A string channel

[RussellSenior]

Andreas, I think you hit the nail on the head.  

There are 1399 channels with 10-14 values each.  

There are 74 string channels.

And each channel contains some properties. 

So, any suggestions on reducing the footprint?  It compresses quite well with 2,527KB compressing down to 119KB.  Does Diadem include a TDM compression function?

Would it be better to use TDMS? 

[AndreasK · Accepted solution]

There is an compression method available that only works for the tdm header.

Option Explicit

dim targetfile : targetfile = "C:\temp\target.tdm"
call DataFileSave("<filename>" & replace(targetfile, "&", "&amp;") & "</filename><headermode>zip</headermode>","TDM")

which is only available using script.

Be aware that this will end up with a aompressed tdm header and a binary tdx file.

But it might help in your place if you really need small headers.

Old DIadem version will also not be able to read those files.

You also will pay with a little additional loading time. Check on your own.

[RussellSenior]

Excellent.  You have always helped me out, and I really appreciate it.  Next time you are in the Metro Detroit area, let me buy you a beer?