New HDF5 VI Library

[Jason_S]

I have been developing a new HDF5 interface library for several months, and have finally managed to get it adequately packaged for distribution. I didn't find the NI-provided library to be sufficiently easy to use, or sufficiently complete for my needs. At this point, the library is essentially complete, and I have been successfully using it in my research for several months. However, it should still be considered beta quality. It is available for Windows and Linux.

Further details and the library itself may be found at:

http://www.me.mtu.edu/researchAreas/isp/lvhdf5/

Please drop me a line, either in response to this post or to the e-mail address listed in the readme with any questions, bugs, suggestions, flames, etc.

I hope that you find the library useful. I'd hate to have created all of those icons for just my own personal pleasure!

Jason

[DFGray]

Thanks for supporting all of us with your efforts!  A couple of comments that may improve your code.

1. The timestamp is a 128 bit fixed point number with the decimal point in the center (64 bits of integer, 64 bits of fraction).  In the NI API (and NI-HWS), this is saved as a custom 128 bit integer using the following algorithm.  The Type is created by taking a NATIVE_LONG and changing the size_in_bytes to 16, the number_of_significant_bits to 128, and the endian to big endian.  On the LabVIEW side, the timestamp is typecast to an array of U32s (it really doesn't mattter, as long as it is an array of integers) before being saved as an attribute.  Set the call library node appropriately.  Simply reverse the process to read it.
   
2. Did you handle the issue of thread safety?  HDF5 for Windows does not have thread safety, so NI added an interface layer to achieve this.  HDF5 for Linux can include thread safety, if you compile it correctly.  Without thread safety, you run a very high risk of corrupting your data and/or crashing LabVIEW/HDF5 runtime (been there, done that - it is too easy in LabVIEW to create parallel processes).  Alterately, the programmer can ensure thread safety through the use of semaphores or a similar mechanism.

Thanks again!  Let me know if you need clarification/expansion on any of the above.

[nealpederson]

Great job Jason! I struggled with HDF5 a couple of years ago for a project
that required all of my data files to be stored in HDF. I was able to get
through my requirements but it was a struggle and in the end it did not
support the latest version of HDF. From a brief look at your website it
looks like you have overcome the problems that I encountered. I look
forward to using your code in the future if another project comes up with
HDF requirements. - Neal Pederson

"Jason S" <x@no.email> wrote in message
news:1155557408159-403618@exchange.ni.com...
>I have been developing a new HDF5 interface library for several months, and
>have finally managed to get it adequately packaged for distribution. I
>didn't find the NI-provided library to be sufficiently easy to use, or
>sufficiently complete for my needs. At this point,&nbsp;the library&nbsp;is
>essentially complete, and I have been successfully using it in my research
>for several months. However, it should still be considered beta quality. It
>is available for Windows and Linux.Further details and the library itself
>may be found at:<a href="http://www.me.mtu.edu/researchAreas/isp/lvhdf5/"
>target="_blank">http://www.me.mtu.edu/researchAreas/isp/lvhdf5/</a> Please
>drop me a line, either in response to this post or&nbsp;to the e-mail
>address listed in the readme with any questions, bugs, suggestions, flames,
>etc.
> &nbsp;
> I hope that you find&nbsp;the library&nbsp;useful. I'd hate to have
> created all of those icons for just my own personal pleasure!
> &nbsp;
> Jason

[Jason_S]

1) I'm using the 64 bit UNIX time for saving the timestamp, and the fractional seconds are lost. For my own purposes, I'm usually just recording "when did I do this experiment" so fractional seconds are not necessary. However, I'm not particularly pleased with the solution. The original thinking was that this would enable other programs (i.e. HDFView) to automatically display the times. However, none of the HDF viewers that I have encountered actually interpret either the 64 or the 32 bit UNIX time, so it doesn't much matter. Future versions will probably switch over to a similar 128 bit datatype, or perhaps a cluster.

2) I handle thread-safety in only the most minimal way, (just like the HDF5 library!). Namely, my library's internals won't be screwed up by multiple threads. However, I make no effort to protect the user from from HDF5's lack of thread-safety, except that the VIs themselves are not reentrant, and so a certain level of protection is inherent so long as calls are not coming from multiple diagrams in different execution systems simultaneously.

However, I don't know the full extent of HDF5's lack of thread safety. With my library, I suppose that it would be possible to delete a dataset while reading it in two different loops on the same block diagram, since this would involve calls made to different library VIs. I can see this might be a problem which I had not previously considered.

So far I have not encountered any problems, but I tend to perform all of my file access from one loop (either the main program loop, or a dedicated I/O loop), which would tend to avoid any thread-safety problems that may exist.

I will look into the issue further for future releases.

[DFGray]

You may actually be OK for thread safety.  By default, call library nodes run code in the UI thread.  This effectively runs the code in a single thread.  If you didn't change it, you will be fine.  However, it also gives you about a large performance hit.

[Jason_S]

A brief update on the thread safety issue. It is possible to build a thread-safe version of the HDF5 library on Windows as well as Linux:

http://hdf.ncsa.uiuc.edu/windows/faq.html#General_Q6

This page seems to refer mostly to the upcoming version 1.8.0, so it's not clear to me whether this will work for 1.6.5. However, 1.8.0 will be fully backwards compatible with 1.6.5.

I think handling the issue at the library is the best course of action, if it is available.

[Jason_S]

The DLL calls are not made from the UI thread in order to avoid the thread switch penalty.

[DFGray]

That reply is worth some stars!  Thanks for the info.  It's about time they did that.

[whit]

I am trying to read an HDF5 3D dataset consisting of 60 frames of 2048X2048 U16 data. The problem is I cannot figure out how to only read in one frame at a time, and when I read all the frames I get an out of memory error. Any help would be appreciated.

Gary

[DFGray]

You didn't say what LabVIEW API you are using, but here is the general idea.  You need to use select_hyperslab on your file 3D dataspace before reading the data.  This allows you to read any portion of your data you desire, up to and including a random set of points.  Make sure your memory dataspace is a 2D array (2048x2048 fixed).  Hopefully this helps.  HDF5 has a pretty steep learning curve.  Hang in there; it is worth it.

Note, this is a very old thread.  You would probably get more response if you started a new one.