Concatenating TDMS Files

[grmume]

Hello,

I'm writing a script in DIAdem 2011 which combines data from multiple sources into a single TDMS-File. Exporting into multiple TDMS files works fine( I can open them in DIAdem) but I'm having trouble combining all files to one TDMS-File. Speed is important, so I tried to avoid loading all data into the dataportal and saving again (this works though).

If I understood correctly it is possibly to directly concatenate the binary contents of the files into a single file. Is this correct? Because using the MS-Dos copy command or using a VBS-Script in DIAdem with Stream-Objects both results in TDMS-Files that I can't open anymore.

The error message says:

"TDS Exception in Intialize: Tds Error: TdsErrNotSupported(53):"

It seems only one of the files produces these problems. When I load this one file and save it again as TDMS I can sucessfully combine them and load the resulting TDMS without errors.

Perhaps the file is fragmented and this leads to the error (the offending file is measurement data)? Is this expected behaviour for fragmented files?

How can i efficiently combine these files? I have to be able to combine multiple hundreds of files ( in pairs. e.g. combine 400 TDMS in pairs to 200 TDMS files) fast and i guess opening 200 files, saving again and combining is rather slow.

Is there an efficient way to solve this task?

As the offending file is actually TDMS-Data from a BLOB-Datafield in a SQL-database another solution might be loading the TDMS directly from the database into the dataportal and saving from DIAdem, instead of dumping the BLOB-data into a TDMS-File on the HDD( i guess this results in a defragmented file as opening and saving resolves the problems).

But I'm not sure how to achieve this. I'm using ADO to interface with the SQL-DB. The code looks approximately like this:

  Set recordSet = CreateObject("ADODB.Recordset")
  recordSet.Source = 'QUERY
  recordSet.open
  Set fso = CreateObject("Scripting.FileSystemObject")  
  'Create Stream object
  Set BinaryStream = CreateObject("ADODB.Stream")
  do
    chunkSize=100
    lngOffset=0
    lngFileSize=recordSet.Fields.Item("Field").ActualSize
     'Open the stream And write binary data To the object
    BinaryStream.Open
    do while lngOffset<lngFileSize
      varChunk=RecordSet.Fields.Item("TDMS_DataInBLOB").GetChunk(chunkSize)
      Call BinaryStream.Write(varChunk)
      lngOffset=lngOffset+chunkSize
    loop    
    'Save binary data To disk
    Call BinaryStream.SaveToFile("path", adSaveCreateOverWrite)
    BinaryStream.Close    
    Call RecordSet.MoveNext
  loop until RecordSet.EOF

 How can I change this code to write the data into the dataportal?

Thanks in adavance.

Best regards,

grmume

[Brad_Turpin]

Hi grmume,

The ADO stream to get the BLOBs to disk is a required first step-- DIAdem won't receive the ADO stream object directly.  It actually works quite well to sandwich separate TDMS files together into one big one with DosCopy commands.  Here is an example of creating and then running a DOS batch file to sandwich small TDMS files into one big one.

Set fso = CreateObject("Scripting.FileSystemObject")        
DosCop = ""
DosDel = ""
FOR i = 0 TO UBound(ChanNames)
  AddFileName = Folder & "\" & ChanNames(i) & ".TDMS"
  IF DosCop = "" THEN
    DosCop = "copy /b """ & AddFileName & """"
  ELSE
    DosCop = DosCop & " + """ & AddFileName & """"
  END IF
  DosDel = DosDel & vbCRLF & "del """ & AddFileName & """"
  DosDel = DosDel & vbCRLF & "del """ & AddFileName & "_index"""
  DosDel = DosDel & vbCRLF & "del """ & AddFileName & ".log"""
NEXT ' i
DosCop = DosCop & " """ & NewFileName & """"
DosIdx = DosIdx + 1
DosFile = DosDrv & "\TDMS " & Right("000" & DosIdx, 3) & ".bat"
DosDel = DosDel & vbCRLF & "del """ & DosFile & """"
Set f = fso.OpenTextFile(DosFile, 2, True)
f.Write DosCop & DosDel
f.Close
Call ExtProgram("cmd.exe", "/c """ & DosFile & """")

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[grmume]

Hi Brad Turpin,

I have already tried concatenating the files with the copy command. I don't have a hex-editor at work but by looking at the filesizes and checking the contents with a texteditor the script seems to work but I can't open the created file.

The script I used looks like this:

private Sub mergeFiles(filePath1,filePath2,outFilePath,bDeleteInput)
  if bDeleteInput then
    Call ExtProgram("cmd.exe","/Q /c copy /b """&filePath1&""" + """&filePath2&""" /b """&outFilePath&""" &del """&filePath1&"""&del """&filePath2&"""")
  else
    Call ExtProgram("cmd.exe","/Q /c copy /b """&filePath1&""" + """&filePath2&""" /b """&outFilePath&"""")
  end if  
End Sub

 To test your script I defined the following variables:

NewFileName="C:\Documents and Settings\username\Desktop\new folder\combined.TDMS"
ChanNames(0)="file1"
ChanNames(1)="file2"
Folder="C:\Documents and Settings\username\Desktop\new folder"
DosDrv="C:\Documents and Settings\username\Desktop\BatchFile"

 It completed sucessfully but the result is the same. I can open the two seperate files(file1.tdms and file2.tdms) but I cannot open the combined.tdms.

Sadly I can't share the original files and I can't use DIAdem to fill the files with dummydata because opening in DIAdem and saving resolves the problems.

Do you have an idea what causes these problems? The possibility to combine tdms-files like that was a big bonus for me...

Can I somehow fill the files with dummydata without opening them in DIAdem? Or tell DIAdem to keep the original file structure but change the values?

Best regards,

grmume

[Brad_Turpin]

hmmmm,

Well, it's been a while since I've done this.  The code that I went back to actually had the ExtProgram() command commented out and used the windows shell object (WSH) instead.  I though that was for synchronous/asynchronous purposes, but perhaps the ExtProgram() didn't work at all-- I don't remember.  Why don't you give this way a shot, this was what I was actually using, and it absolutely worked:

Set WSH = CreateObject("WScript.Shell")

   :          :

On Error Resume Next
Call WSH.Run("""" & DosFile & """", 0, FALSE)
On Error Goto 0

Brad Turpin

DIAdem Product Support Engineer

National Instruments