Advanced TDMS reading

[Intaris]

I am tearing my hair out here....

I have written a TDMS file using the Advanced Write function with buffering and in synchronous mode.  The data is there, I can see it.

I have written 4 channels of data (each with 520 points) in interleaved manner.  I also have a second set of channels (also 4 channels with 520 data points each) which I want to perform a SUM on.  I want to read in both sets of channels, add them and write them back to the original set of channels (To have a running sum in my file). I know that if I read in the data using the advanced palette, my data will actually be interleaved (0,1,2,3,0,1,2,3,0,1,2,3) but that's OK, it doesn't matter because BOTH blocks are like that so the result is still correct.

It works fine if I read ALL data at once (i.e. "Set read position" followed by reading all 2080 datapoints, adding and re-writing).  My problem arises when I try to do the work in chunks.  The final data sizes could be in the range of gigybytes, so I need to have some kind of partitioning going on.

When I try to read X points at a time (Set Offset 0, read X, Set Offset X, read X, Set Ofset 2X and so on) I get errors on the second iteration (Setting offset X).  Why is that?  I even tried setting X = Sector size but that didn't help either.

[Intaris]

I have it.

The "Set offset" function takes into account that the data is interleaved (so in my case, setting Ofset 520 will actually try to set Offset 2080 in pure binary terms as I have 4 interleaved channels) whereas the Read function doesn't care a bit about the interleaved status.  I need to tread my "Read" and "Offset" values very differently.

So I need to divide my offsets by 4 and it works.

That just cost me over half a day to find out.  Efficiency, yay!

[Intaris]

TDMS may be the death of me.

Now I see other problems.  I have a portion of code which returns no data event hough I just wrote that data to file.  I set a breakpoint just before that code.

1) if I abort the probram then and there, I see all the data in the file.

2) If I open the file without aborting the program, the data is NOT there.

Even adding a "Flush" immediately before the breakpoint does not help.  Could it be that I am forming the TDMS file reference?  Does this reference need to be maintained (i.e. do the output references change at any time?

Again, I'm working with buffer mode advanced TDMS functionality.

LV 2012 SP1

[YongqingYe]

Why you need to write the data values back to the TDMS file? I would not recommend write them back by using TDMS Advanced API since you have to understand TDMS file format and the layout of the TDMS file very well. Is it possible to create a new TDMS file with the sum data values?

[Intaris]

I do not want to create a second file.  I want to be able to maintain the entire data set ina single file as this file will later be accessed by third parties.  Spreading the data over multiple files is not an option.

I am fully aware of the internals of the TDMS file.  Running the same code on a previously written TDMS file works fine.  It's just that running it on a file which is in the process of being witten seems to not work properly because the TDMS functions seem to perform some buffering which is not affected by the "Flush" command.  That's what it looks like to me at least. If I close the TDMS file and runt he code on it at a later time it works fine.

I think it may be linked to the fact that I have NOT disabled buffering when opening the TDMS file.  Whatever buffering is going on int he background seems to be interfering with my code.  I might have to switch it off, but to be honest, it feels kind of buggy.

[Intaris]

I found my problem.

The write position does not reset when defining a new set of channel meta data.  Once I have done my processing, I need to set the file write position to the end of the file otherwise my subsequent data overwrites the old data.  This is why my offline files were behaving signifcantly differently than the "online" files.  There was no overwriting in the "offline" files.

Is there any sure-fire way to set the write position to the end of the file without knowing the actual order/structure the file was written?

[Intaris]

Eacht ime I think I've solved a problem, another one pops up in its place.

I seem to be missing channel properties.  I cannot find the Property "NI_ChannelLength" for most of my channels (although they definitely ARE there and the properties even show up in the TDMS explorer).

I have even tried getting the entire property list for the channel but all Isee is "name".  No Datatype and no Length.  If I close the file then and there the properties show up no problem in the TDMS explorer.

Is soemone playing a practical joke on me?

[Intaris]

It really seems that if I don't actually CLOSE the file and REOPEN it, the properties for certain channels are not always present.  This is a big problem for me.

My workaround is to place a CLOSE followed immediately before another OPEN just before the problems otherwise would occur.

As mentioned previously a  FLUSH does NOT help.  This feels very much like a bug.

[johnsold]

It does sound buggy. The help file for TDMS Flush clearly states that all buffers will be written to the file. That should include any attributes as well as data.

My guess is that NI has optimized the file writes for speed. To do so they end up keeping some things like the attributes and other properties in buffers as long as possible. It seems that this is at least a plausible explanation for your observations.

Something they might be able to do would be to access all the buffered data as well as what has already been written when a Read occurs. How feasible that is may depend on where the buffers are and how accessible they are to other VIs.  There could also be issues with race conditions if both reading and writing are occurring simultaneously.  Something which might work well for your particular situation may create worse problems in other cases.

Lynn

[YongqingYe]

The simplest way to get to the end of the file is, first to get the size of the file and then use Set File Position.