07-07-2015 09:29 AM
I ran your file with a 430MB file and these are the returned load and save times.
This save time is a lot faster than my original method but as you can see it still takes over half an hour.
Here's some further information.
DIAdem: 14.0.1Service Pack (32-bit)
USI: 14.0.2
As for the File information, the channel number and length varies from file to file. Usually there are anywhere from 10 to 30 channels.
07-07-2015 10:59 AM
I tried to reproduce locally with a NAS and watching your numbers I strongly
believe that it is related to the up and downstream of your datashare connection.
20000 kbit/s
1390 kbit/s
Be aware that I slightly changed the example because the
DataFileMove method works different than I expected.
I tried to create an script to reproduce.
In my system with a NAS and a 400MB file.
Channels: 30
Size : 400,02 MB
Save : 14,9813 s
Load : 13,1988 s
Option Explicit 'Forces the explicit declaration of all the variables in a script. dim targetPath : targetPath = "\\192.168.2.2\public\out.tdms" dim filesizeinmb : filesizeinmb = 400 dim nrOfChannels : nrOfChannels = 30 dim nrOfValues : nrOfValues = filesizeinmb * 1024 * 1024 / 8 dim chLength : chLength = CLng(nrOfValues / nrOfChannels) MsgLineDisp "Generate data" LoopInit data.root.clear dim grp : set grp = data.Root.ChannelGroups.Add("group") dim i : for i = 1 to nrOfChannels loopinc i * 100/nrOfChannels Call ChnGeoGen("/ch" & i,i,i*100,chLength,"") Next loopdeinit MsgLineDisp "Delete fie if already exists" if FileExist(targetPath) then DataFileDelete targetPath end if MsgLineDisp "Save to target file" stopwatchreset 1 SaveTdmsLocalAndMove targetPath dim dT1 : dT1 = stopwatch(1) dim realFileSizeInMb : realFileSizeInMb = DataFileSize(targetPath)/1024/1024 MsgLineDisp "Load file" data.Root.Clear stopwatchreset 1 LoadTdmsFileImediately targetPath dim dT2 : dT2 = stopwatch(1) if FileExist(targetPath) then DataFileDelete targetPath end if MsgBox "Channels: " & nrOfChannels & VBCRLF &_ "Size : " & FormatNumber(realFileSizeInMb,2) & " MB" & VBCRLF &_ "Save : " & FormatNumber(dT1, 4) & " s" & VBCRLF &_ "Load : " & FormatNumber(dT2, 4) & " s" sub SaveTdmsLocalAndMove(byVal targetPath) dim fileType : fileType = UCase(right(targetPath, len(targetPath) - instrRev(targetPath, "."))) dim targetPathStr : targetPathStr = left(targetPath, instrRev(targetPath, "\")) dim targetFileName : targetFileName = right(targetPath, len(targetPath) - instrRev(targetPath, "\")) dim tmpTargetPath : tmpTargetPath = TmpDrv & targetFileName call DataFileSave(tmpTargetPath, fileType) call DataFileMove(tmpTargetPath, targetPathStr) end sub function LoadTdmsFileImediately(byval srcFile) dim oldBulkDataLoadingMode : oldBulkDataLoadingMode = BulkDataLoadingMode BulkDataLoadingMode = eBulkDataLoadingImmediately on error resume next DataFileLoad srcFile LoadTdmsFileImediately = err.number on error goto 0 BulkDataLoadingMode = oldBulkDataLoadingMode end function
07-07-2015 01:04 PM
Running that exact code here are my reults,
If I am correct this would imply that my issues are related to the up and downstream of the connection.
07-08-2015 01:13 AM
It is hard to believe that you can save 400MB in 1.6 s
to a network share. So I am not sure what we are seeing here.
The read througput looks like 100 mbit/s network.
Would it be possible to share one of the tdms files.
Maybe upload it zipped to ftp://ftp.ni.com/incoming/
07-08-2015 01:00 PM
The 1,654s is 28 mins for the 400 MB file. I'm using american notation for my , and . in the numbers. sorry if there's any confustion there.
Also unfortunately I can't share the data due to company policies.
07-09-2015 12:58 AM
Yes the number imply that the saving time is just related to the upstream.
The DataFileMove method uses basic Windows functionallity.
So copy the file with Explorer should use the same amount of time.
What changes do apply to the file?
07-09-2015 07:57 AM
All I do in this step is changing the existing file name. However in the future it will be adding channels, properties, etc. for data analysis.
07-09-2015 09:07 AM
Rename can be done using
DataFileRename
potentially without having to move the file completely.
If you just want to edit/add properties.
CreateDataFileHeaderAccess
might help you without having to read the file completely.
But if it comes to add new channels ... you have to move the file and writing will be slow.
07-09-2015 09:09 AM
ok, thank you so much for all of your help!