Showing results for 
Search instead for 
Did you mean: 

NI-HWS hdf5 file format: reading from Matlab

Background: I'm storing ~22MB files acquired at 250MS/s to disk using the NI-HWS VIs. The ultimate "number crunching" program will be in Matlab and will process perhaps hundreds of thousands of files for a particular data collection run.

So I save my data and then using the Matlab hdf5info function (we'll call the output of this function fileinfo), figure out (after much trial and error) that I retrieve my data set via (you ready?):


It seems like supplying a Group and Waveform name doesn't change where the data is stored in the hdf file. Is this *the* way to get at my data or am I missing something?


P.S.  - Since there's no other way to store the t0, dt, offset and gain waveform info retrieved from the niScope card (I use this in displaying the data), I save these to the waveform attributes X Axis Start, X Axis increment (those two make sense), Y Axis Minimum, Y Axis Maximum (these make less sense, but there's no other better attributes and at least it works). Again, does this seem the best way to save that information?
0 Kudos
Message 1 of 4
You pretty much got it.  However, the to, dt, offset, and gain are automatically saved in the HWS file.  You don't have to do anything extra.  Changing the title of a channel changes that attribute of the waveform, but does not change the file structure.  This makes reading the file easier in the long run.

Now, where is all this data?  The internal structure of an HWS file is similar to SCPI-DIF, but can potentially hold a lot more information.  I would encourage you to download the HDF5 browser, HDFView, from NCSA so you can trace the structure in your file.  Here is some info to get you started.

Each set of data is stored in a top level group titled wfm_groupn where n is an integer.  Inside this group are four groups - traces, axes, vectors, and id.  The traces group contains the traces.  A trace is a line which can be drawn on a rendering device.  A trace can be anything from a function drawn with equally spaced x-axis points to an arbitrary XY data set.  The axes group contains the axes contained in the traces.  Two axes are necessary for each trace.  Axes can be either explicit (contain a number of data points) or implicit (described by a formula).  If explicit, the data is stored in the vectors group.  The id group gives identifying information about the storing program and what version of the file format is being used.  A simple example will help illustrate.  The HWS file for a single waveform will look something like this (ignoring most of the id info)

    \timestamp - I128 version of LabVIEW timestamp
      \name - "Time"
      \type - 1 (type can be 0 - explicit, 1 - implicit, or 2 - combination/arithmetic operation)
      \start - 1.0  (this is relative to the timestamp given in the id section, if any)
      \increment - 1.0
      \name - "Velocity"
      \type - 0
      \explicit_vector - 0 - may not be present in some cases since data_vector provides the same information
      \data_vector - hard link to the data in the vectors group
      \scale_coef - {a0, a1, ... } - dataset containing the polynomial scaling coefficients.  a0 is the offset;  a1 is the scale; etc.
        \name - Channel 1
        \x_axis - 0
        \y_axis - 1
        \data - { ... }

Note that LabVIEW timestamp data types are stored as signed, 128bit, big-endian integers.  In reality, they are a 128-bit fixed point number with the decimal in the center (64-bit integer, 64-bit fraction).  The timestamp is the number of seconds since Jan 1, 1904, 12:00 midnight GMT (start of the year).

If you need more information, let us know.  I realize this was once over very lightly.
This account is no longer active. Contact ShadesOfGray for current posts and information.
0 Kudos
Message 2 of 4

Thanks, DFGray. It definitely seems you're the expert regarding hdf5/NI-HWS/high-speed/large data set acquisition on these forums!

It looks like in the datasets I'm saving there's only 4 subgroups under wfm_group0 (i.e. no 'vector' group) when I open them in Matlab. (perhaps it's because I'm only passing it a vector of I8s --see below-- and not a true waveform) But the hardlink under axis1 does the trick.

So you say the HWS automatically saves t0, dt, gain & offset? How does it know this information if all it's being passed is a 1D vector of I8s? Perhaps it doesn't and that's why I don't have the vectors subgroup? The niScope VI passes out a vector of data and a 'waveform info' in a separate data entity, but unless I combine them into a waveform data type (which adds seemingly unnecessary overhead and copying in memory) I guess I don't see how HWS can keep track of those entities unless I manually add them to the file.

So, if manually adding them to the file is the way to go, are there any VIs under HWS (I don't have access to my development computer atm) that allow you to insert the data directly?

Right now it's not too big of a problem since I'm only using t0, dt, gain & offset to display the data in LabVIEW. Once it gets opened in Matlab all they care about is that 1D vector of data.

0 Kudos
Message 3 of 4
I assumed you were using a waveform, not a raw array, which would automatically store the t0 and dt.  I have no excuse for the scaling coefficients other than lack of coffee Smiley Happy.

However, to get things in the right places, check out the niHWS Store Analog Waveform  It contains an example of how to store the scaling and timing information.  You will have to open the write VI to get the t0 and dt info.

Sorry about that.  Let us know if you need more help.
This account is no longer active. Contact ShadesOfGray for current posts and information.
0 Kudos
Message 4 of 4