DIAdem

cancel
Showing results for 
Search instead for 
Did you mean: 

Large File Save

Solved!
Go to solution

I ran your file with a 430MB file and these are the returned load and save times.

Untitled.png

This save time is a lot faster than my original method but as you can see it still takes over half an hour.

Here's some further information.

DIAdem: 14.0.1Service Pack (32-bit)

USI: 14.0.2

 

As for the File information, the channel number and length varies from file to file. Usually there are anywhere from 10 to 30 channels. 

0 Kudos
Message 11 of 19
(2,309 Views)

I tried to reproduce locally with a NAS and watching your numbers I strongly

believe that it is related to the up and downstream of your datashare connection.

 

20000 kbit/s
 1390 kbit/s

 

Be aware that I slightly changed the example because the

DataFileMove method works different than I expected.

 

I tried to create an script to reproduce.

In my system with a NAS and a 400MB file.

 

Channels: 30
Size    : 400,02 MB
Save    : 14,9813 s
Load    : 13,1988 s

Option Explicit  'Forces the explicit declaration of all the variables in a script.

dim targetPath : targetPath = "\\192.168.2.2\public\out.tdms"


dim filesizeinmb : filesizeinmb = 400
dim nrOfChannels : nrOfChannels = 30

dim nrOfValues : nrOfValues = filesizeinmb * 1024 * 1024 / 8
dim chLength : chLength = CLng(nrOfValues / nrOfChannels)

MsgLineDisp "Generate data"
LoopInit
data.root.clear
dim grp : set grp = data.Root.ChannelGroups.Add("group")
dim i : for i = 1 to nrOfChannels
  loopinc  i * 100/nrOfChannels
  Call ChnGeoGen("/ch" & i,i,i*100,chLength,"")
Next
loopdeinit

MsgLineDisp "Delete fie if already exists"
if FileExist(targetPath) then
  DataFileDelete targetPath
end if

MsgLineDisp "Save to target file"
stopwatchreset 1
SaveTdmsLocalAndMove targetPath
dim dT1 : dT1 = stopwatch(1)

dim realFileSizeInMb : realFileSizeInMb = DataFileSize(targetPath)/1024/1024

MsgLineDisp "Load file"
data.Root.Clear
stopwatchreset 1
LoadTdmsFileImediately targetPath
dim dT2 : dT2 = stopwatch(1)

if FileExist(targetPath) then
  DataFileDelete targetPath
end if


MsgBox "Channels: " & nrOfChannels & VBCRLF &_
       "Size    : " & FormatNumber(realFileSizeInMb,2) & " MB" & VBCRLF &_
       "Save    : " & FormatNumber(dT1, 4) & " s" & VBCRLF &_
       "Load    : " & FormatNumber(dT2, 4) & " s"


sub SaveTdmsLocalAndMove(byVal targetPath)

  dim fileType : fileType = UCase(right(targetPath, len(targetPath) - instrRev(targetPath, ".")))
  dim targetPathStr : targetPathStr = left(targetPath, instrRev(targetPath, "\"))
  dim targetFileName : targetFileName = right(targetPath, len(targetPath) - instrRev(targetPath, "\"))

  dim tmpTargetPath : tmpTargetPath = TmpDrv & targetFileName
  call DataFileSave(tmpTargetPath, fileType)
  call DataFileMove(tmpTargetPath, targetPathStr)
  
end sub

function LoadTdmsFileImediately(byval srcFile)

  dim oldBulkDataLoadingMode : oldBulkDataLoadingMode = BulkDataLoadingMode
  BulkDataLoadingMode   = eBulkDataLoadingImmediately
  on error resume next
  DataFileLoad srcFile
  LoadTdmsFileImediately = err.number
  on error goto 0
  BulkDataLoadingMode = oldBulkDataLoadingMode

end function
0 Kudos
Message 12 of 19
(2,292 Views)

Running that exact code here are my reults,

MoveTime.png

If I am correct this would imply that my issues are related to the up and downstream of the connection.

0 Kudos
Message 13 of 19
(2,286 Views)

It is hard to believe that you can save 400MB in 1.6 s

to a network share. So I am not sure what we are seeing here.

The read througput looks like 100 mbit/s network.

 

Would it be possible to share one of the tdms files.

Maybe upload it zipped to ftp://ftp.ni.com/incoming/

 

0 Kudos
Message 14 of 19
(2,267 Views)

The 1,654s is 28 mins for the 400 MB file. I'm using american notation for my , and . in the numbers. sorry if there's any confustion there. 

Also unfortunately I can't share the data due to company policies.

0 Kudos
Message 15 of 19
(2,246 Views)

Yes the number imply that the saving time is just related to the upstream.

The DataFileMove method uses basic Windows functionallity.

So copy the file with Explorer should use the same amount of time.

 

What changes do apply to the file?

  • Adding properties
  • Channels
  • Changing values
  • do the tdms_index files exist on your share
0 Kudos
Message 16 of 19
(2,230 Views)

All I do in this step is changing the existing file name. However in the future it will be adding channels, properties, etc. for data analysis.

0 Kudos
Message 17 of 19
(2,216 Views)

Rename can be done using

DataFileRename

potentially without having to move the file completely.

 

If you just want to edit/add properties.

CreateDataFileHeaderAccess

might help you without having to read the file completely.

 

But if it comes to add new channels ... you have to move the file and writing will be slow.

Message 18 of 19
(2,203 Views)

ok, thank you so much for all of your help!

0 Kudos
Message 19 of 19
(2,200 Views)