Hello Akshay,
DIAdem offers two features to deal with this situation directly.
- "Selective Opening" of files - this feature will only load a subset of your data channels from a file, based on your selection. This can reduce load times because only the data required for the operations you wish to execute in DIAdem will be loaded. This feature is especially useful when your file contains hundreds of signals, or if you have very long channels. This feature can be found in the "File Open ..." dialog in the NAVIGATOR and can be scripted.
- DIAdem also offers the ability to use files straight from disk. Go to "Settings: DIAdem Settings" and select the "Loading Behavior" setting. Make sure the setting there says "Load bulk data when changing channel data". The help system has a section on "Loading Bulk Data" in the NAVIGATOR help section that gives a lot more background.
In general, it helps to have data local (vs. a network drive) when processing large numbers of files or large files (or both). Network access adds considerable time to the loading process, so it might make sense to temporarily copy data to a local drive (or USB 3.0/eSATA attached external drive) to improve performance.
Also, if your files are large, you should use the 64-bit version of DIAdem and have sufficient memory (I recommend 8GB or more) to make sure DIAdem works as efficiently as possible.
For use cases such as yours, we will introduce a server based tool next year to have multiple processes run in parallel on a server - no more sequential running of dozens or hundreds of files on a client - you will be able to start a process on a server and it will automatically parallel process all your files with no need to have to wait on your client. Keep a look out for the "TDM server suite" over the next few months.
I hope this helps,
Otmar
Otmar D. Foehner
