Inbuilt Support for Hierarchical Data Format (HDF5 etc)

[MartinMcD]

I would like the option to save data from LabVIEW using the Hierarchical Data Format. This would make it much easier to share data with Matlab and other programs.  

This is a real pain point for us at the moment and drives many of our engineers away from using LabVIEW, quite rightly. The tdms import dll for Matlab is slow and the 3rd party .mat export toolkits for LabVIEW are not sufficiently developed or only licensed for single-seat environments.

[MartinMcD]

I've just found a similar idea from 2010 that is still listed on the idea exchange as 'new'. Can anyone from NI please comment on whether this is being looked at?

[X.]

I don't think you should expect NI to mess with open formats. In fact, I don't think you should even want it.

If you read the other thread carefully, you'll see that the only input from NI engineers was "have your tried our TDMS format?" and "which functionality would you want TDMS to have so that we could make you forget about HDF5?"... Do you want me to explain why the insistence?

There is an open project going on (referred to in the other thread): this should get your support. Considering how much traction is needed to have NI correct bugs in their primitives or even DLL-based VIs (read: it's a completely opaque decision process, assuming they even register your bug report), a buggy native support by NI would be your worst nightmare.

[Darren]

Duplicate of http://forums.ni.com/t5/LabVIEW-Idea-Exchange/Suggest-that-LabVIEW-support-HDF5/idi-p/1206363

[GregSands]

So X., you think NI should have developed its own proprietary image format* rather than using PNG or TIFF or JPG files?  I think integrating low-level support for standard formats (of which I think HDF5 certainly is one) opens up far more possibilities than it restricts.

* Actually NI did develop the AIPD image format for the Vision module, but I don't know if anyone anywhere has actually ever used it.

[MartinMcD]

X, I have read the other thread carefully and I do understand your concerns but I still find NI's lack of support for HDF puzzling and very frustrating.  Inbuilt support for the format in LabVIEW would be far preferable to the situation we have now.

I assume the current lack of support is partly for commercial reasons to keep people within the LabVIEW ecosystem, but it achieves the opposite.

[X.]

@GregS: AIPD is exactly my point. I started using the IMAQ Toolkit (as it was called before it was purchased by NI) when there was no support for other image formats. I looked into TIFF libraries because the Vision Toolkit does still not fully support it (if you care to check it in detail). I would bet that a lot of options in the other two formats you mention are not supported as well. Ergo, yes, I think an Open Source project to develop support for these formats would be a good idea.

@MartinMcD: I am not saying NI should not support it. I am just saying I am not sure that it is in our best interest compared to a community-based project, because they simply have other priorities than their customers' (not saying that they do not care about their customers, but mostly those who pay big bucks, understandably).

[GregSands]

X., I agree completely regarding TIFF support.  I've written my own VI library wrapped around (some of) libTIFF so that I can read/write 3D TIFF images, but I think even that only highlights the limitations of not having integrated support because I have to manually copy data out of a Vision Image in order to do so.  In terms of data files, the DataPlugin framework does go some way towards to supporting other formats, but it is still far more difficult than the fully integrated support for TDMS.

All I'm saying is that I think HDF5 is a worthy candidate for that same level of of integration (as TDMS) because it would allow sharing scientific data with almost any other software package - the point of the original poster here.  And I'd say exactly the same about TIFF (though it's probably one of the most difficult formats to support well as it is so fractured).  The only other format that comes to mind that is poorly supported (or unsupported) in LabVIEW but very widely used elsewhere is the DICOM format.  I know NI are probably expecting customers to be locked in to NI products for all their work, but especially in the research community that is almost never the case.

[X.]

The problem is that the list of "universal" file formats is endless. What is worthy to you will be worthless to me and reciprocally.

As long as NI doesn't put obstacles to the development of open source add-ons to their products, I don't see a problem with that.

As far as images are concerned, the way they are stored in memory is insane and has been since the beginning of the IMAQ toolkit, so I'd argue that a complete rewrite would be needed to have efficient image handling (check page 1-7 of this document to understand the beauty of borders and padding), even in C.