TDMS Read fails on channels of different lengths

[sth]

LV experts....

It seems that there is an error in the TDMS read functions.   I have a TDMS file generated by a built executable, this file has one group of 9 channels.  The 5th channel is data taken at a higher rate (ie 20 MHz) compared to the date in the other 8 channes (1 Mhz).  The 1 Mhz channels have 110,000 points, the 20 Mhz channel of course has 2,200,000 points.

If I read this using the TDMS Read function passing it an array of all 9 channel names.  This executes and returns an array of 9 waveforms with correct Y0 and dt values and the correct .  However channel 4 (the 5th channel, the long one) has garbage in the data for all points past the the 110,000 in the other channels stop at.  This garbage is like numbers -10^304 and zero.  The first 110,000 points are correct.

The file itself is not corrupted.  If I read the file in a for loop 1 channel at a time and build the array of wave forms I get all 9 wave forms and the correct data of varying length.  A trivial example of reading the actual data file is attached with the data file (I hope the boards will let this be attached.)  The top plot has the corrupted data of the all-in-one read and the bottom has the correct data from the for loop version.  The arrays waveforms on the left show that the corruption occurs precisely at array location 110,000 where the other arrays 

Yes, the data file is too long for the forums.  You can get the simple example program and data file at

TDMS error.zip

Any advice or if this is a real error.  I see nothing in the documentation that I am doing anything wrong.  The fact that it returns a valid array length and no error code leads me to believe it is a LV bug.

[Hazen_berg]

Hello sth,

I would attempt to run LabVIEW as the Administrator.  Sometimes, low level TDMS functionality requires these priviledges.  It is rare, but we have seen cases arise in the past.

This is just off the top of the head, so I will go ahead and look more into this.  Let me know if this helps.

Cheers,

[sth]

Yes, I am an admin account on the laptop where I ran this.  Interesting concept but that is a really weird failure mode.  That would be a very bad bug.

I have observed this on my laptop under Mac OS X and then sent it to a friend who also as an admin under windows saw that same thing.

[Hooovahh]

I don't think this has anything to do with account privlates. I suspect this behavior is because the TDMS read function generally will return a 2D array of values, where columns are channels and rows are samples if you read multiple channels and multiple samples.  In this case as you probably know, LabVIEW 2D arrays have the same number of rows, for all columns.  I think the TDMS read does this same operation on a waveform and just turns it into a 1D array of waveforms afterwards.

For example try instead of reading the channels using the waveform data type, just read it as a double.  You'll see the returned values are correct for the Y axis of your channels, but you have that same issue of not having the same number of samples in all channels.

The workaround is simple enough, read your 1D array of waveforms for channels that are at one sample rate, and do another read for the channels in the other sample rate.

[sth]

Hoovah,

That is probably a fine workaround.  However this means that the documentation is broken and the fact that the long waveform is returned not as a truncated waveform but as a long wave form with uninitialized data is a very very non-LabVIEW.  In the TDMS file channels are stored as linear data streams not as 2D arrays so there shouldn't be a problem reading it as an array of waveforms.  In fact the file format seems to be designed for that rather than a 2 D binary array.  THere is a property kTocInterleavedData for the data but I don't think it is for storing interleaved arrays.  The waveform data type is the default format.s

See http://www.ni.com/white-paper/5696/en/

I maintain that it is a cross platform bug and that it violates several LV tennants of returning bad data or bad lengths of uninitialized data.  Fortunately I am not time limited and reading the arrays as a default format in a for loop will suffice and the savings of determining how many waveforms are of the same length and then reading just those in a single read is more complicated for the time savings.

However, I looked at reading into a 2D array which is also not kosher for generic array of 1 D waveforms.  It seems that it reads the first 110,000 points into a 2D array and then adds 2 millon zeros to most of the data and corrupts the data in the long array again back to -10^304 values.

This can't be the designed behavior!

[Hazen_berg]

What data type are you using?  The values you are describing are oddly close to the lower bound of the double data type.  This would occur if the data you are taking exceeds the higher limit, at which point the values roll to the lower limit and start rising again.

[sth]

---

@Hazen_berg wrote:

The values you are describing are oddly close to the lower bound of the double data type.

Yeah, I noticed.  That is why I am saying the data is corrupted.  If you look at the example I posted and read the data, all the values are between 0.15  and -0.15.  Nowhere near - 10^308. This is what I get if I look at it in the TDMS data viewer.  This is what I get when I read the waveforms one at a time.

If I do a multiple channel read either into a 1 D array of waveforms (the default) or into a 2 D array of doubles I get the corrupted data.

Lets see if I can say this concisely.  The TDMS functions are broken, it is a bug that the routines are returning false data contrary to the documentation and API.

[Hooovahh]

---

@sth wrote:

Lets see if I can say this concisely.  The TDMS functions are broken, it is a bug that the routines are returning false data contrary to the documentation and API.

---

I don't think anyone is arguing that it isn't a bug, and doesn't need to be fixed.  Maybe I'm a bit optomistic but I imagine some TDMS applications engineer is running a test and figuring out what is happening, and what can be done to fix it.  

[sth]

Yes, I opened an SRQ on this and it has been referred to R&D.  It is "not expected" behavior.   I am just startled that this hasn't bitten anyone else.  I guess it is just one of my strengths to try my first simple TDMS read function and get what looks like a long standing fundamental error.  Fortunately, I have a work around and I will document the weirdness in my code if some trys to optimize it in the future that it can be tested.

[emmazheng · Accepted solution]

Hi sth,

Thanks for your findings. The issue you reported does exist for reading multiple jagged scaling channels once. (TDMS Read on multiple jagged unscaling channels works fine.) The .tdms file you provided including scaling data.

I can provide two workarounds. The first is what you already mentioned, read each scaling channel inside a loop. The second is that still read all channels once, but rank all channels by length in descending order.(The first is the channel with maximum length; the last is the channel with minimum length.)

Regards,

Jie

NI R&D