Large File Save

[Artemis125]

Hi all,

I've been working through scripting in DIAdem, and i've encountered a big time sink. Some of the files I work with are upwards of 200MB and these files load within a minute or two however when I save the data it can take 20+ minutes to save. I was wondering if there's any way to speed up this process or if we should change our data aq processes to make smaller data files. The files are Initially Read Only and are then edited and saved under a new name. Let me know if there's anygthing I can do.

Thanks,

Sadie

[Pesmith8]

Sadie,

What format are the output files? 

Is it saving the files to a network drive, or to a local drive.?

TDM or TDMS both write substantially faster than that.  ( I have used files over 2x that size)

Paul

[Artemis125]

I'm using TDMS files, saved onto a network drive.

[Pesmith8]

Hi Sadie,

I can think of one way that would take that amount of time to save.  That is if the channel type was Text and not numeric. (and it was a long text channel.)  Really all channels should be only numeric, with the properties holding any text values,  If you had something that was like a text value that varies, it would store best if the values could be made into enumerations, Where the text that corresponded to the number were stored in channel properties. This allows channels to have numbers in them.  

Paul

[Artemis125]

I don't have any text channels. All channels are numeric.

[Otmar]

Hello Sadie,

Could you please try saving the same file to a local hard drive, and timing that vs saving to a network drive.

I assume that saving to the network is slowing this down, but I would like to eliminate other causes by timing the save time to a local hard drive ...

Thanks,

     Otmar

[Artemis125]

Otmar,

I just tried saving to the local hard drive and it took less than a minute to save. I do need to save these files onto a network though. Any ideas how to speed up that process?

Thanks,

Sadie

[Otmar]

Hello Sadie,

I am not sure why writing the file to a network vs. writing it to a local directory has such significant time impact, but one way to work around this is to write the file to your local disk and then copying it to the relevant network location.

I'll ask someone from R&D to check this thread and hopefully they can provide some insight and advice on how to accelerate writing to a network drive ...

     Otmar

[AndreasK]

Could you please add some additional information:

• DIAdem/USI version (can be found in DIAdem About box)
• What content does your file have (Number of groups / Number of channels / length of channels)

How is the behavior if you store a TDM file to your share instead of an TDMS file.

[AndreasK · Accepted solution]

One effect that may influence is the fac that some time of loading the file might be contained in the save time.

The following script contains some effects.

• BulkDataLoadingMode causes by default that bulk data is loaded on access. In the script the file is completely loaded.
• SaveTdmsLocalAndMove saves the file to temp folder and moves it afterwards.

If the share is connected via DSL the difference in up and downstream speed may influence the result.

(20 seconds is a lot of time for 200 mb tdms file. So it is potentially no local windows share)

Option Explicit

dim srcPath : srcPath = "\\server\share\example.tdms" 
dim targetPath : targetPath = "\\server\share\exampleOut.tdms" 

data.Root.clear

if FileExist(TargetPath) then
  fileDelete(TargetPath)
end if

dim txt : txt = ""

stopwatchreset 1
call LoadTdmsFileImediately(srcPath)
txt = txt & "Load " & stopwatch(1) & " s" & VBCRLF

stopwatchreset 1
call SaveTdmsLocalAndMove(targetPath)
txt = txt & "Save " & stopwatch(1) & " s" & VBCRLF

MsgBox txt


sub SaveTdmsLocalAndMove(byVal targetPath)

  dim fileType : fileType = UCase(right(targetPath, len(targetPath) - instrRev(targetPath, ".")))

  dim tmpTargetPath : tmpTargetPath = TmpDrv & "\SaveTdmsLocalAndMove." & fileType
  call DataFileSave(tmpTargetPath, fileType)
  call DataFileMove(tmpTargetPath, targetPath)

end sub

function LoadTdmsFileImediately(byval srcFile)

  dim oldBulkDataLoadingMode : oldBulkDataLoadingMode = BulkDataLoadingMode
  BulkDataLoadingMode   = eBulkDataLoadingImmediately
  on error resume next
  DataFileLoad srcFile
  LoadTdmsFileImediately = err.number
  on error goto 0
  BulkDataLoadingMode = oldBulkDataLoadingMode

end function