DIAdem shows data incorrectly

[Brad_Turpin]

Hi Steve,

Let's distinguish clearly between the 3 schema choices you'll need to make:

    A) The TDMS footprint that will be used to store the data

    B) The appearance of that TDMS data when indexed by the DataFinder

    C) The Data Portal footprint when selected data is loaded into DIAdem

The 4 points in my previous post referred to C), what DIAdem needed to have in the Data Portal to show the curves you want to see in DIAdem.  That doesn't mean you need to store the data that way in TDMS.  There could be a custom load routine that pulls the data from your preferred TDMS schema into DIAdem and maps it to DIAdem's needed Data Portal schema.  I could even build a "post-processing" DataPlugin that would map your preferred TDMS schema to a suitable one for querying for use by the DataFinderr.  Note that the DataFinder doesn't really appreciate XY Time/Data channel pairs any more than you do, so now we're talking potentially 3 different schemas: TDMS files, DataFinder, DIAdem.

1)  The problem you have with TDMS storage is that one waveform per data channel can't handle the inactive regions.  Period.  You either need multiple waveform channels per data channel, or you need to have a Time channel per data channel.  If you're worried about space, you don't need to use the full 16 bytes per datetime value, you could save a DBL Time channel or an I32 Time channel (with elapsed seconds) and store the start time as a property.  Again, LabVIEW and TDMS do not offer you the ideal storage container for your active/inactive burst data.  I really think you need to pick one from these two options, then we can make the best of it downstream.

2)  Note that I carefully said insert ONE row of NaNs.  What you describe is technically a possible solution for 1), but I'm assuming it's too expensive on disk space and haven't considered it seriously (although it would make everything DELIGHTFULLY simple).  What I'm describing here is the Data Portal content that DIAdem needs, and DIAdem needs only ONE row of NaN's between active and inactive data rows.  DIAdem does NOT need you to fill in all the inactive time with multiple NaN rows, as many as the sampling period demands.  This is the advantage of using a Time channel per Data channel, you can specify the exact XY pairs that were acquired.  DIAdem needs the one row of NaN (and could just be one NaN in either the Time channel or the Data channel), in order to plot the "gap" in the curve during the inactive period.  Otherwise DIAdem will draw a line across the inactive period, which you don't want.

3)  An implicit Time channel is a feature in DIAdem and TDM files that is not offered in LabVIEW and TDMS files.  It stores a "y = mx + b" linear formula with 3 values.  This makes the memory footprint of the implicit Time channel just 20 bytes, no matter how long it is.

     DBL = Start Value

     DBL = Increment Value

     I32   = Number of Values

4) Close.  I'm actually suggesting C1_time, C1, C2_time, C2, C3_time, C3...  And remember, this is only for the Data Portal schema, this does NOT need to show up in the TDMS file (though it's one of your choices), and it also does NOT need to show up in the DataFinder (even if it is what you choose for the TDMS files).  Having the C1_time channel before the C1 data channel all but guarantees that dragging the two channels together will put the C1_time channel on the X axis.  Having them the other way around would all but guarantee that the C1 data channel ended up on the X axis, which is not what you want.

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[Brad_Turpin]

Hi Steve,

I thought of another option for your TDMS storage that you may prefer.  It's surprising it took me this long to think of it, since it's what you were asking for from the beginning.  The disconnect is that you were asking for waveform features that don't exist, but there's really no reason we can't implement exactly what you want in the TDMS channels, as long as we're committed to a custom viewer.

1)  You could save your multiple bursts to the same TDMS channel, but each time the acquisition state changes, you could add a set of 3 TDMS channel properties:

    BufferStartTime1  = 11/20/2015 09:00:15
    BufferDeltaSecs1 = 0.1
    BufferValsCount1 = 36000
    BufferStartTime2  = 11/20/2015 10:00:15
    BufferDeltaSecs2 = 0.01
    BufferValsCount2 = 6000
    BufferStartTime3  = 11/20/2015 10:01:15
    BufferDeltaSecs3 = 1.0
    BufferValsCount3 = 3600
    BufferStartTime4  = 11/20/2015 22:32:59
    BufferDeltaSecs4 = 0.1
    BufferValsCount4 = 374689

    BufferCount = 4

The above example shows a single Data channel stored as a single TDMS channel where the sampling rate changed each time, and there was an inactive period between Buffer3 and Buffer4.  I'd recommend that if there is no change in the sampling rate and no inactive period between actual DAQ buffers that you update the common Buffer... properties rather than create a set for each DAQ buffer.  TDMS supports overwriting the previous property value by just writing the property again as if you never had before.  I also advise you to create and maintain a BufferCount property so that the custom readers can quickly know how many sets of time information are stored as properties for this TDMS channel.

For the sake of clarity, let me list the property data types I'm imagining:

BufferStartTimeN  = DateTime

BufferDeltaSecsN = DBL

BufferValsCountN = I32

Brad

[CoastalMaineBird]

So, how would that last plan work, in terms of the user dealing with it?

I can see how it would be interpreted, but I don't want the user to have to deal with segments and rearranging and all that.

Can I simply load one of those files into DIAdem and have it show up correctly (without any finagling)?

Will DataFinder understand this interpretation if it is asked for the value of CO2 at 12:34:56.789, or What time was the maximum NOx value, and such questions?

[CoastalMaineBird]

Understand that I'm a noob with DIAdem (in case that's not obvious), so I don't exactly know about a "custom viewer".

Does that mean that anybody who wants to read one of these files has to install some custom translation / intrpreter thingy?

I have 30+ people who will be reading them, do they all need to install this gizmo?

[Brad_Turpin]

Hi Steve,

By custom viewer, I meant that whatever TDMS reading software (DIAdem, LabVIEW, ...) you use will need to have a mechanism to "unpack" the buffer times in a way that it can handle.  The problem is that a single waveform can't handle either the changing sampling rate or the inactive periods.  That's too bad, because a waveform is a standard vehicle that has support in most TDMS clients, especially LabVIEW and DIAdem.  Since you need to do more than what a waveform can offer, we need to pack that information into either a Time channel for each Data channel or a series of properties attached to each Data channel.  Either way, the TDMS client will then need to view that non-standard time information, and that will take dedicated code (VIs, VBScripts).  Now for the case of DIAdem and the DataFinder, I'm offering to help you define and create that code, and we can also embed it into the user experience so that no "extra" actions are necessary.  DIAdem supports custom load events, which we could define for your N DIAdem clients, such that when these TDMS files are loaded, the necessary time information unpacking occurs automatically.

Before we go any further, please be aware that DIAdem datetime values are still stored in a DBL (like LabVIEW's used to be), resulting in an accuracy of 0.1 milliseconds.  You listed 1.0 millisecond accuracy in your datetime example value, and I want to make sure you aren't surprised by this limitation later.  For long term logging, this is usually not a problem, so I had not brought it up yet.

Now, the key concept to understand about the DataFinder for your current question is that it only stores the properties of the Files. Groups, and Channels.  It does NOT store the data values in the channel arrays-- these are only located in the indexed data files.  This means that you can not query data values from channels, you can only query properties from Files, Groups, and Channels.  So if you want to query the maximum value of a particular channel, you need to store that maximum value as a property before the TDMS file is indexed by the DataFinder.  If you want to find particular features within the values of one or more channels, then you need to load the corresponding TDMS files into DIAdem or LabVIEW and perform the feature detection.  If you then save the results of your feature detection to various File, Group or Channel properties and then reindex the data file, then the DataFinder can absolutely query those properties to present you all the Files/Groups/Channels that contained that feature.

Brad Turpin

DIAdem Product Support Engineer
National Instruments

[CoastalMaineBird]

Since you need to do more than what a waveform can offer,

--- Not that it matters now, but I still don't see that as true. The WAVEFORM structure itself offers EXACTLY what I needed.

I came in writing several Waveform structures, each having a START TIME and a deltaT.

That describes the data the way I need to see it.  It's just that DIAdem chooses to ignore all but the first of those.

It's not the waveform structure that is insufficient.

DIAdem datetime values are still stored in a DBL

---  Even if I feed it a TIMESTAMP (16 bytes), it truncates it to a DBL?  How very 1980s.

1 mSec is sufficient for now.  We have to do some high-speed stuff separately, but that will be crank-angle related, and may not be in TDMS.

it only stores the properties of the Files. Groups, and Channels.

--- That I knew, but didn't show it in my examples.  You can't really search for the time of the peak NOx value, unless we record that as a property sometime before DataFinder gets hold of it.

[Brad_Turpin]

Hi Steve,

Just for the record, it is not DIAdem that is ignoring the additional "to" and "dt" waveform properties, it is the TDMS Write.vi.  When you save the second buffer of Channel1 to the TDMS file, you have two choices.  If you save it to a new "GroupName/ChannelName", then the "to" and "dt" properties are saved correctly and you get a new TDMS channel.  If you save the second buffer to the same "GroupName/ChannelName" that you saved the first buffer to, the "to" and "dt" properties are NOT saved, and the "Y" array is concatenated to the end of the existing TDMS channel, and NO new channel is created.  You are correct that the waveform wire that holds each buffer of your data for a given channel has all the properties you need.  But the moment you send that new waveform buffer to a TDMS waveform channel that already exists, you lose the "to" and "dt" properties of all but the first waveform written to that TDMS channel.  The TDMS channel is designed to hold exactly one waveform, it is not designed to hold a series of waveforms that form one composite curve with gaps and varying acquisition rates.

You still have the option of sending each of your waveform buffers to a new TDMS channel-- that was one of the two options I initially listed for task 1).  In that case we could teach the DataFinder to see only one waveform channel per physical channel, and we can teach DIAdem to morph the various component buffer waveform channels into one Time and Data channel pair during loading.  All of your options for 1) result in a TDMS storage that will look wrong in the TDMS viewer, but this is a TDMS limitation, not a DIAdem limitation.  I guess I confused the issue by describing it as a waveform limitation, since the brown wire in LabVIEW is not imposing the limitation, the TDMS Write.vi is the culprit.

Brad Turpin

DIAdem Product Support Engineer

National Instruments

[CoastalMaineBird]

You still have the option of sending each of your waveform buffers to a new TDMS channel-- that was one of the two options I initially listed for task 1).  In that case we could teach the DataFinder to see only one waveform channel per physical channel, and we can teach DIAdem to morph the various component buffer waveform channels into one Time and Data channel pair during loading.  

--- Is there an example of such a beast around?  I'd like to see how the data file looks before and after installing this widget.

If there's source code for the widget, so much the better.

--- Is the source code in DIAdem script, or LV code, or what?

--- I would like for the user to not even notice that such stitching-together is happening.

All of your options for 1) result in a TDMS storage that will look wrong in the TDMS viewer, but this is a TDMS limitation, not a DIAdem limitation.

My backfilling option seems to work in both scenarios (albeit not for sample rate changes) but I don't really care about correctness in the VIEWER - that's just a development tool, and I can adjust my head around whatever weirdnesses are there.

[Brad_Turpin]

Hi Steve,

I will have to develop the "widget" after you send me a sample TDMS file with the storage schema you decide on.  The "widget" will be coded in VBScript.  DIAdem itself is coded in C++ for the most part, though I never see its source code myself.

I agree that the goal is for the user to be unaware that there is any morphing happening between the TDMS files, the DataFinder index, and the DIAdem Data Portal.  If you choose either two of the three options for 1) which do not employ a Time channel, then the astute user will notice that Time channels suddenly appear in the DIAdem Data Portal, but they won't know how that happened, and hopefully they'll be too busy looking at the automatically generated graphs to care.

You're right, for the case of filling in NaNs for every inactive sampling period in a single waveform per channel, the TDMS viewer does indeed show you the curves correctly (assuming a constant sampling rate).  I'm still guessing this is not going to be your choice, since the inactive period could conceivably be long enough to waste too much waveform memory stuffing it with NaNs.

Brad Turpin

DIAdem Product Support Engineer

National Instruments