TDMS Extract Segments

[Hooovahh]

So I'm learning a lot about the TDMS file structure using this document.

http://www.ni.com/white-paper/5696/en/

My goal is to be able to break up a TDMS file into its segments, or rather break up a file that has 2000 segments, into 100 files with 20 segments in them each.  Opening a TDMS file into a hex editor I can see all the segments, and I can save the first 20 Segment on its own into a new file and it opens just fine in the Excel import.  But if I try to take the second 20 Segment (without the first 20) and save it away the Excel Import craps out with a code 6.  I tried just taking segment 21 and it also dies in a similar way.  Attached is the original fragmented file which should have 40 segments, the file split into the first 20, the second 20 and a file with just the 21st.

So my question is this.  What am I doing wrong in the split process?  Why does segment 21 on its own create an invalid TDMS file?

EDIT: I also tried deleting the index files forcing them to be recreated.

[Wayne.C]

Hooovahh,

Try using the TDMS Read function to pull data out of the file instead of hex editor.  I've attached a chunk of code I use to do something similar. It's based on something I originally found on the NI website.  Please excuse the lack of refinement.  This code is used to convert a bunch of 24 hour datalogs to Excel.  Each .TDMS file gets converted to a coresponding Excel file.  The number of worksheets in each workbook is determined by user input.

[emmazheng]

Hi, 

It's not quite common for splitting a TDMS file by segment numbers. I just try to explain your two questions. 

1."What am I doing wrong in the split process? "

As your description, I think when you create the TDMS file containing the second 20 segments or only the 21 segment, you simply copied the binary from the original file. The binary for the second 20 segments(and the 21 segment) is relative with the binary in previous. The binary you copied for latter segments is lack of some parts(details explained in question 2), so can not compose a valid TDMS file format. But the binary for the first 20 segments is intergrated for a valid TDMS file format. That's why you can just successfully open the TDMS file containing the first 20 segments.

2."Why does segment 21 on its own create an invalid TDMS file?" 

As a conclusion, the binary of "21st Segment.tdms" is lack of the parts below in Meta Data:

  1) The root object and group object ;

       From the binary snapshot of "21st Segment.tdms", we can see there is no binary for root object and group object in Meta Data part(Address 0x1c ~ 0x3d).

  2) Details of raw data index for the channel object.

       The snapshot shows Raw Data Index part in Meta Data is 0x 00 00 00 00(Address 0x36~0x39), that means the raw data index of the object in this segment exactly matches the index the same object had in the previous segment. But for an independent TDMS file, this part can not be omitted. The missing part is identical with the part of Address 0x6b ~ 0x7e in the snapshot of the 1st segment.

If the above parts are added in Meta Data, the next segment offset and raw data offset in Lead In(Address 0x0c~0x13 and 0x14~0x1b in the binary of "21st Segment.tdms") should also be updated.

The binary of "21st Segment.tdms":

The binary of the 1st segment(within the red frame):

[Hooovahh]

Wow thanks.  A lot of useful information in there.  I fear that what I want to do will be much more complicated then I first planned and as a result I may give up on it.  Adding the root group is very easy, and in fact can be the first segment of every new file.

The difficult part is the Raw Data Index part.  You see if I decide to split at segment 20 (or any segment) I will likely have a segment that states to use the previous segment information (Raw Data Index being 0x00).  I could simply add the empty information to that segment like you suggested, but this will be for a specific Group/Channel pair, and the next segment maybe for a different Group/Channel pair which also states to refer to the previous values for that object.  So if I took segments 20 thorugh 39, do I need to check for each segment containing a 0x00 as the Raw Data Index and add empty information, and update the Lead In offsets?

An alternative that I haven't got to work is to again make the first segment contain all the needed information for all Group/Channel information.  I'm not done testing but as before I am creating TDMS files that can't be opened.  Any other tips are greatly appreciated.  Thanks.

[emmazheng]

Hi,

My explanation for your further questions is as below:

1."Do I need to check for each segment containing a 0x00 as the Raw Data Index and add empty information, and update the Lead In offsets?"

If any of the latter segments has a new pair of Group/Channel which is not included in previous segments, you need to do this.

2."Alternative: make the first segment contain all the needed information for all Group/Channel information"

I think this is a feasible approach. I can give you the specific steps:

a. Copy the first four segments binary from "First 20 Segments.tdms"

b. Add the copied binary at the beginning of "Second 20 Segments.tdms". If there is "Second 20 Segments.tdms_index" file, delete the index file. Then you can open the updated "Second 20 Segments.tdms" file successfully, but each channel containing samples [0,5,6,7,8,9] (not [5,6,7,8,9] as expected).

c. Delete the 4 segments(the 5th ~8th) with sample value "5" (0x05 00 00 00 or 0x35) in "Second 20 Segments.tdms". Modify the sample value "0"(0x00 00 00 00 or 0x30) to "5" in the first 4 segments(the 1st~4th). Then delete the index file, and reopen the modified file. You will find each channel contains samples [5,6,7,8,9].

The modified "Second 20 Segments.tdms" is attached.

NI R&D

Jie

[Hooovahh]

Again thank you for your advice it has been helpful.  But I am not trying to split this specific file, I am trying to split any TDMS file into N files.  Doing this programatically has become a challenge.  As I said before performing a TDMS open and list contents I can create a new TDMS file that is just the structure, and I can make that the first segment for all new files, but adding the other Raw Data Index for each object is where I am having difficulty. 

I have attached what I have been able to do so far.  The main file is Main Perform TDMS File Split.vi.

To finish the program I need to fix the Raw Data Index issue and at the moment I'm thinking for each segment, determine what the object being written is, then determine if it has a 0x00 for Raw Index Offset.  Then determine if the Raw Index Offset has been written to a segment previous to the current segment, for the current object.  If it hasn't create a new segment which writes the Raw Index Offset.  This seems tedious but is the only way I know of at the moment to make a valid TDMS file.

[emmazheng]

Hi

I think you already know what to do for splitting a TDMS file to multiple files by the number of segments.

Just to clarify if your goal is only to split a large TDMS file, a simple way is to read samples from each channel and then write the content to multiple small TDMS files separately. (Relative topic in the forum)

Jie

[deppSu]

I think you can extract the segment where you plan to do the split and make this segment a temperary tdms file. If this file can be opened with TDMS Open without any error thrown out, then it's reasonable to split at .

[Hooovahh]

I know that I can open, read a subset, and create new files, but I was hoping I could do this without even performing the TDMS open.  I realize my example uses the Open, but my idea is if I have a very large TDMS file (like multiple GB) then the Open will even take a while, so I was hoping I could split the file, perform operations on each file, then combine them back.  The operations and combine are quite easy but the split is what I didn't know if I could do.

---

@deppSu wrote:

I think you can extract the segment where you plan to do the split and make this segment a temperary tdms file. If this file can be opened with TDMS Open without any error thrown out, then it's reasonable to split at .

---

My point is that the TDMS Open will throw errors so the split wasn't done right.

[mcduff]

---

@Hooovahh:

---

I know this is an old post, but it is pretty awesome! Thanks @Hooovahh for all your contributions to TDMS. There is a possible edge case, that may not be relevant now or in the future, but I thought you may want to know as the VI here is similar to the one in your Tremendous TDMS Package.

Your Get TDMS Segment Offsets will fail if you have a big-endian TDMS file. I do not think this is common, but thought I would mention it. You need to read to the ToC after the "TDSm" tag and look at the bit that marks endianess. See this. The ToC is always little endian, so you can do your file read in little endian, or make a switch depending on the binary read. Below shows the work-around but you will probably be more clever than myself. Attached is a big-endian file that you test if desired.