Explicit Timestamps in TDMS files

[Jed_Davidow]

I occaisionally use TDMS files.  But I keep encountering one significant problem and I wonder if there is a good way to deal with it.

Typically the systems I deal with record non-deterministic data (glommed from several RS232 sources, etc).

A typical situation is a cluster of several DBLs that needs to be written at 1Hz or so.

Since the data is non-deterministic, the waveform type is not appropriate (The Timestamps are not perfectly even).

So I use an explicit Timestamp channel.  The problem is that I cannot seem to write the TS channel (as a TimeStamp data type) and the DBL data at the same time.  Inevitibly, I end up with a file with more timestamps than DBL data.

I have tried various methods to link this data (the best thing I have found is to take the TS data and massage it into a DBL and then record that with the process data.  This enables me to validate the TS later on, but also kind of renders useless the whole point of this exercise in the first place.

Any good ideas out there?

[Hooovahh]

Correct me if I'm wrong, but you have some data coming in at a changing rate.  Attatched is a quick example that writes two sets of data, each with its own time stamp for each sample.  Then it will read back the data and display it on an XY graph.

[Jed_Davidow]

Thank you for the code.

This is basically what I have implemented in the past (although, normally I have reduced the data to one set of data and one timestamp to go along with all values, but yes, occaisionally I end up with multiple Timestamp arrays).

The issue is that each piece of data is written to the file in a different call than with it's corresponding TimeStamp.

Looking at the first half of your example, if the data is written without error, but then there is an error writing the timestamp, you now have a serious problem.

Of course, if the error is catastrophic, the issue is moot.  But if it's a recoverable error, now you have a file with more data than timestamps.  And IF your error handling is not up to snuff, you might end up recording more data after this point.  When you inspect it you will be short some amount of TimeStamps with no way to know where the error occurred.  Of course, we can record the errors into the file (or another), but this doesn't work if there are file write errors!

What I want to do is to be able to write the TS with the data, atomically.  So that there is one call, and if anything goes wrong NONE of the data is written; or possible some way to write all the data+TS into the buffer, but be able to clear the buffer (without flushing to the file) if something goes wrong.

[Hooovahh]

I understand your point about errors happening with the data which cause time stamp data to not match it.  Which is why I agree a primative that writes both data and time for each sample is a great idea.

Having said that let me say I have been using this technique for a while and the only errors I've had are in the initial testing.  What kind of error could I have with writing data to a TDMS, that I couldn't resolve before considering the application complete. 

Remember I agree with you, I'm just saying I don't have a problem writing them my self seperately.

[deppSu]

I think if you wire the TS write right after the corresponding data write with error terminal connected and in case the data write failed the TS write will not do either as long as the error is not cleared before the TS write. 

[Jed_Davidow]

The problem is when the first write is good and the second fails.

[johnsold]

Convert your timestamps to Doubles and include them with the array of data?

Lynn

[Jed_Davidow]

That has been the best solution.  It would be nice if they could be native though, but I think this is what has to happen.

[FrankHS]

It would be great if they had a way to store a 2d array in TDMS other than makeing each column a channel.

[GerdW]

Hi Frank,

two notes:

- I think you need to explain why and how you want a 2D array saved in a TDMS file

- if you think you and others may profit from this suggestion you should post an idea in the LabVIEW Idea Exchange!