DIAdem shows data incorrectly

[Brad_Turpin]

Hey Steve,

You know, you'd get everything you want in LabVIEW if you created a new waveform channel every time you wrote a data buffer, then your buffer start datetime value and your buffer acquisition rate would be recorded each time, and if you graphed each of those individual buffer waveforms (with the same color), they'd show the gaps you're looking for in the inactive regions.  You could even have a post analysis routine in LabVIEW or DIAdem to put the data together into a master Time and Data channel set that included a NaN value for each inactive region.

The main trouble you're having is that you're appending each new buffer to an existing waveform TDMS channel by using the same Group name and Channel name in the TDMS Write VI, and as Hooovahh mentioned, this only adds the values to the DBL array, it doesn't update the start datetime property or the sample period property that latched permanently with the first waveform write.  If you use a different Group name/Channel name combination for each buffer, you'll capture all the data acquisition information you're trying to.  Of course, this will also create a boatload of Channels, so that might not be feasible.

DIAdem makes the issue worse for you by ignoring the start datetime waveform properties (sorry).  I can help you build a VBScript to create DateTime channels and plot the curves automatically, but first you need to decide on a way to capture all the data acquisition information that currently is not making it into the one waveform channel.

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[CoastalMaineBird]

 it doesn't update the start datetime property or the sample period property that latched permanently with the first waveform write. 

That's what I have surmised from all this testing, and that's what seems broken, but you're saying "it's not a bug, it's a feature". 😉

You know, you'd get everything you want in LabVIEW if you created a new waveform channel every time you wrote a data buffer, then your buffer start datetime value and your buffer acquisition rate would be recorded each time, 

Not really, unless I STILL go thru the rigamarole of creating a time channel for each track and pairing them up correctly.

The first graph in this thread shows that a channel that STARTS after the others will be shown incorrectly, even if it's START TIME is set different.

Of course, this will also create a boatload of Channels,

Well, I expect this active/inactive switching or sample rate switching to be relatively rare, so it wouldn't be a boatload.  But I have to cover that case, and current behavior is unacceptable.

But having multiple channels for the same signal would create other problems, it seems like.  What is the maximum for this channel? What is the standard deviation? WHat is the correlation between this channel and some reference signal?  I have to answer all those, and having the data in two or more channels muddies up the water.

Having a post-processor doesn't help, I don't think.  My latest version has a "backfill" procedure, which recognizes the gap and fills it with NaN, which seems like it solves those analysis problems. I don't think that helps the sample-rate switch, but we may end up disallowing that while a file is open.

My customer doesn't like any of this, so I'm now looking at recording a TIME trace for each possible sample rate (1,2,5,10,20,50,100,200,500,1000) and finding a way to associate each signal channel with one of those time traces.  Hopefully, that will cover the gaps issue sufficiently.

[CoastalMaineBird]

OK, there is STILL something that I am misunderstanding, because it STILL appears broken to me.

Today's attempt creates a TIME channel for each of my possible sample rates (1,2,5...1000).

Forget changing rates for the moment - I just want DIAdem to get the START TIME right.

Here is the recording, as shown in TDMS Viewer:

At 5 sec in, Chan 2 goes OFF (inactive)

At 7 sec in, Chan 3 comes ON (becomes active).

No muss, no fuss, it looks correct.

HOWEVER

If I load the same file in DIAdem, and painstakingly associate each Y signal with the 100 Hz TIME signal, it STILL gets it wrong:

You can see the TIMEBASE 1 Channel occurs every 1 sec, the TIMEBASE 2 Channel occurs every 0.5 Sec, etc.

So... Why does chan 3, which is recorded with a START TIME of 5 sec after the others, get shoved to the front?

I thought I understood yesterday, that the FIRST waveform header is the one that gets remembered and assumed valid for the rest of the channel signal, but it's ignoring that as well.

So... how do I get it to tell me the truth?  

Do I REALLY have to go through that converting each waveform to a Time channel every time I load a file?

Do I REALLY have to go through the pairing up this signal and that timebase?  Is there an automatic way?

Is there some way to record the data in a way that DIAdem will show it correctly?

[CoastalMaineBird]

Here you can plainly see that the START TIME for Chan 3 is correctly recorded in the file as 10:37:21, but DIAdem plots it starting at 37:16 or so.

[CoastalMaineBird]

OK, so NOW he reads the help file:

I wonder why it says "MIGHT".  Does it decide on a whim to discard or not ?  Maybe it does it if the user is STEVE, but not for JOE?

I wonder why it doesn't include the "dT" part - it seems like it always discards that for subsequent waveforms as well.

[Hooovahh]

---

@CoastalMaineBird wrote:

I wonder why it says "MIGHT".  Does it decide on a whim to discard or not ?  

---

I'm not NI, but I can think of a reason why it might not discard it.  Lets say I wrote to a channel an array of doubles.  Then I write to that same channel a waveform.  In this case it won't discard the start and dt information because it has never been writen as a property before.  I think a more clear explaination is it will write the start and dt time from a waveform, if one has not been written to that channel yet.  If it has it will discard this information and keept he t0 and dt information that is already associated with that channel.

Each channel has have, at most, one dt and one t0, just like a scalar waveform.

[Brad_Turpin]

Hi Steve,

Wow, that LabVIEW Help content is odd, and I agree with your complaints.  The dt is ignored as well and so is the offset-- only the first start time and offset and dt are recorded, but for each subsequent buffer write to the same channel, all those properties are never written again, to my knowledge.

The only way to plot vs. a datetime X axis in DIAdem is to use a datetime X channel.  When you plot a waverform vs. a numeric or datetime X channel, all waveform properties are ignored.  That means that you need to transfer all the waveform properties to the datetime channel you use as your X channel.  The "Numeric Channels <=> Waveform Channels" function in the ANLYSIS panel does this for you when you select the "Absolute time reference" radio button at the bottom.

Yes, you have to go through all these steps in DIAdem to see what you want.  In fact, if each of your channels can start at an individual datetime, then you could in principle need a datetime channel for every data channel, even if all the data channels had the same sampling period.  I continue to offer my assistance in automating these steps with a script, once you have decided how to store all the data acquisition information you need in your TDMS file.  If you decide to pad inactive periods with NaN values, or if you decide to store a time channel for each data channel in TDMS, then DIAdem will graph that data easily.  If you decide to store multiple waveform buffer channels for each physical channel, then we'll need to stitch those together afterwards or graph them in the same color.

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[CoastalMaineBird]

The dt is ignored as well and so is the offset-- only the first start time and offset and dt are recorded

--- I don't know what "offset" you're referring to - a Waveform only has START TIME, deltaTIME, and the Y[ ] data, unless you want to talk about attributes.

The only way to plot vs. a datetime X axis in DIAdem is to use a datetime X channel. 

I tried that, but it didn't work the way I thought either. 

I created a DateTime channel that started at the FILE start (let's say 13:00:00.000).

I paired up a signal channel with that DateTime channel.

I *-THOUGHT-* that if the Signal channel had a START TIME of 13:00:05, that it would recognize that the signal started at sample #50 (say) of the DateTime channel, and it would plot it starting later.

But no.

It appears that I have to have a DateTime channel that STARTS at the same time as the signal channel.

Which means that every channel that comes ON will need a new, different DateTime Channel.

But that is STILL not enough, if a given channel goes OFF, and then back ON.

In such a case, it looks like I STILL have to "backfill" the gap with NANs.

This is an absurd amount of data to correct something for which all information is ALREADY in the data file - each waveform carries a START time, if it just would stop ignoring it.

I suppose the separate DataTime channel WILL allow a change in sample rate : If I change the DateTime channel at the same point that the Signal rate changes, then they still should track.

I continue to offer my assistance in automating these steps with a script

I appreciate that, but until I decide which way to go, there's no point in automatiing it.

My customer has 100,000 files, now in a private binary format, which need to be converted to TDMS.  They might make another 100k files in the next 10 years.  So you can see how adding this much fluff is painful.

Anyway, thanks for your thoughts.

[Brad_Turpin]

Hi Steve,

In LabVIEW versions past, the waveform wire had a "to" property/component that was a DBL.  I just looked now and see that in LabVIEW 2014 (the one version I have installed) the "to" property on the cluster has been converted to a datetime, so now I understand the confusion.  Under the hood, the old DBL "to" cluster property is always written to the waveform property "wf_start_offset", whereas the new datetime "to" cluster property is always written to the waveform property "wf_start_time".  DIAdem 2015 automatically includes the  "wf_start_offset" property in its numeric X axis graphing, but DIAdem 2015 ignores the "wf_start_time" property because datetime X axis graphing of waveforms is still a missing feature.  This is, in fact, a top support issue, which I fully expect to see resolved in a future DIAdem version, but it won't happen soon, and that doesn't help you in your current task.  I just thought I'd explain what I meant by the "offset" waveform property.  I seem to rememeber versions of LabVIEW where both the DBL "to" property and the datetime property were available from the waveform cluster, but that seems to have gone away.

You seem really committed to getting a workable solution in DIAdem.  Most customers who experience this degree of frustration with the current feature set simply decide to find a different solution outside DIAdem.  Since you need to represent active and inactive data periods, and since you need to support multiple scan rates during the same acquisition, I'd recommend that you aim for the following data schema in DIAdem, regardless of how you acquire the data or what it looks like when LabVIEW initially saves it.

1)  DateTime Channel for each Data Channel, sharing a name prefix, such as "Data ChannelX Time"

2)  Insert one NaN row into the Time and Data Channels for each inactive period-- this will show the gap on the graph

3)  If your Data Channel has only one scan rate and no inactive periods, you could use an implicit Time Channel to save memory

4)  Place the Time Channel directly above the Data Channel in the Data Portal, for easier ad hoc XY drag&drop graphing

I am happy to help you map your customer's binary data to this format directly with a DataPlugin, or indirectly from an intermediate TDMS file, whose format you can freely choose.  I iam also happy to help create a VBScript to automatically make XY graphs from this Data Portal content.

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[CoastalMaineBird]

You seem really committed to getting a workable solution in DIAdem.  

--- my Customer is.  They need the ability to search through data files and extract info - Which tests last week had CO2 levels above 5%, or how many tests run by operator "Fred" failed validation, or whatnot.  I thought TDMS and DIAdem would make that easy. 

Most customers who experience this degree of frustration with the current feature set simply decide to find a different solution outside DIAdem.  

--- That has indeed crossed my mind.

Since you need to represent active and inactive data periods, and since you need to support multiple scan rates during the same acquisition,

--- Changing the scan rate midstream is not an absolute requirement.  It's possible with my DAQ stuff now, but I'm talking with the customer about maybe disallowing that.

--- Disallowing ACTIVE / INACTIVE changes is a different matter. I don't think they want to discard a whole test (hours of data, fuel, operator time) just to turn on one more channel.

I'd recommend that you aim for the following data schema in DIAdem, regardless of how you acquire the data or what it looks like when LabVIEW initially saves it.

1)  DateTime Channel for each Data Channel, sharing a name prefix, such as "Data ChannelX Time"

--- That's really expensive.

Given that a TimeStamp is 16 bytes, that raises the bar to 24 bytes per sample per channel, instead of 8.  Yuck. Double Yuck.

A file might have 400 channels, for two hours long, most channels at 10 Hz, some at 500 Hz. Triple Yuck.

And the user STILL has to pair them up manually, it seems (I haven't figured out how to do that automatically).

2)  Insert one NaN row into the Time and Data Channels for each inactive period-- this will show the gap on the graph

--- That's what I call "backfilling":  when a channel comes ON, I "backfill" with NaNs, one per sample, from the last sample time (or beginning of file) up to the current time.

--- I'm doing that now, with Waveform Data (no TIME channels).  It doesn't help the sample-rate issue, but it covers channels that start late, and that come and go.  The price is just the NaNs in the gaps, not a whole separate channel.

3)  If your Data Channel has only one scan rate and no inactive periods, you could use an implicit Time Channel to save memory

--- Not sure what an "implicit" time channel is.  The Waveform model seems to work for this.

4)  Place the Time Channel directly above the Data Channel in the Data Portal, for easier ad hoc XY drag&drop graphing

In other words, I should write the channels C1, C1_time, C2, C2_time, C3, C3_time - is that what you mean?

I don't want to have to write time channels, but if I have to, this might make it easier to grab them for plotting.

I am happy to help you map your customer's binary data to this format directly with a DataPlugin, or indirectly from an intermediate TDMS file, whose format you can freely choose.  I iam also happy to help create a VBScript to automatically make XY graphs from this Data Portal content.

Thanks, I will probably take you up on that at some point.  For now, we need to decide what features we actually REQUIRE, and what we can live without.