Variant in DLLs

[Jason_S]

Hello,

I'm working on an update to my decaying HDF5 library. In order to increase efficiency and generally have it be "good," it is necessary to work with Variants in the DLL. This presents a problem. Has NI published gotten around to publishing the portions of the LabVIEW API necessary for working with Variants? If not, would it be possible to get some basic functionality published. I really need two functions:

1) Convert from variant to type descriptor and a pointer or handle to the LabVIEW data.

2) Create new and/or modify and existing variant to contain a new type descriptor and point to new data (pointer or handle, as appropriate).

To preempt the suggestion, yes, I could convert into a flat string and TD, pass them out of the DLL, and then use unflatten to variant. However, this is highly inefficient in both memory and speed.

I can work with either 16 or 32 bit type descriptors.

Thanks,

Jason

[WearWolf]

Hey Jason_S,

We do not publicly document how Variants are structured in LabVIEW. Therefore we cannot provide information about how the Variants are structured.

An alternate architecture would be to have multiple DLL functions, one for each of the data types. Then you could have a case structure in LabVIEW that calls a different DLL function depending on which data-type is being used.

Hope this helps.

[rolfk]

---

@WearWolf wrote:

Hey Jason_S,

We do not publicly document how Variants are structured in LabVIEW. Therefore we cannot provide information about how the Variants are structured.

An alternate architecture would be to have multiple DLL functions, one for each of the data types. Then you could have a case structure in LabVIEW that calls a different DLL function depending on which data-type is being used.

Hope this helps.

---

That's very sad! The official (documented) external code interface in LabVIEW has been practically unchanged since about version 3.0 of LabVIEW with very little changes since (PostLVUserEvent() being the only real addition since). Of course the external code interface is not something the majority of LabVIEW developers would use and I also understand that NI has little interest in supporting novice C programmers in how they can use that interface. But much of the power of LabVIEW is basically unaccessible from external code nowadays without starting to reverse engineer the existing but undocumented APIs.

Something like official Variant API access might actually prompt me to drop the drive to support some of my libraries even for rather old LabVIEW versions! Now the use of an external library has absolutely no technical reason to not make that library loadable in old LabVIEW versions, since the API that is documented and officially usable in the external code reference existed as way back as LabVIEW 5 already, except the LVPostUserEvent() which was added in LabVIEW 7.

I don't expect  a detailed technical document of how the Variants are implemented in LabVIEW just a minimally commented header file with the LvVariant... API calls available in the external code interface. Your solution of creating an API export for every possible datatype in a library is not only a huge development and long term maintenance overhead considering the clumsy polymorphic interface LabVIEW has at this point. Also it makes adding new datatypes unneccessarily complicated as each datatype causes changes to both the LabVIEW VI library as well as the external C code. Also if your VI library uses already Variants in order to not have to use the limited polymorphic functionality, you end up converting back and forth between datatypes for every external library call.

Please find a way to make that information officially available. I understand and am fully fine with, if there will be a big watermark across every page "No technical support for this" although I would hope that doesn't cause any legitimate bug reports to be directed automatically to /dev/null.

[Jason_S]

I'd like to echo every thing Rolf said. I couldn't have written a better reply, but that won't stop me from trying 🙂

In my particular case, WearWolf's suggestion is laughable. Now, I don't say this to insult WearWolf--he couldn't have known. The essential problem is that HDF5 can, like LabVIEW, describe nearly any data format. So, if I have a dataset (think array) containing clusters made up of a string, a float, and a subarray of clusters of strings and floats, it can do that. As the user of my library, I might want to read a dataset into LabVIEW without knowing what type was in it. The only way to handle that in LabVIEW is to bring the data in as a variant. At that point, the user can then inspect the variant and make decisions on it. One can imagine that I can't possibly write a separate DLL for every single data type that once can image. What I can (and do!) do is translate between HDF5s data type descriptors (datatypes) and LabVIEW's type descriptors. Once I have a LabVIEW TD, I can see that the HDF5 data is converted into a LabVIEW format (handles and all that).

By the way, we have essentially the same problem in G. Polymorphic VIs are great--sometimes. They're great if you can a priori (ooh--italics) determine all of the possible datatypes that a VI will need to interact with. If not, then you need another option. That's one reason (certainly not the only one) why the File I/O primitives, for instance, are primitives, and not VIs. They have to handle every possible data type. Variants in G go a long way towards helping that problem. Not far enough, but a long way. To close the loop we need to be able to read and create variants in C so that we can interact with other libraries.

Jason

[DFGray]

I can't change the National Instruments policy about publishing variant details (and the internal format has changed a couple of times in the last few years) but I can offer you a possible solution.

Since LabVIEW is a strictly typed language, you really cannot do anything with data unless you know what it is.  HDF5 is completely self describing, but so is a LabVIEW variant.  However, you can't use a LabVIEW variant, other than passing it around, unless you have a wire of the right data type to convert it to something "real."  There is another primitive, the typecast primitive, which works in much the same way, but is a bit more dangerous to use, and is not self-describing.  However, it will give you the functionality you want.  I used a variant of this method to support 64-bit integers in LabVIEW 6.1 ten years ago when I wrote a very old LabVIEW HDF5 library.

HDF5 gives you data by reference, so the data type it thinks it is giving you does not have to match the data type in LabVIEW, they just need to be the same size.  For example, if you are fetching a 128-bit integer from HDF5, you can set the data type in the LabVIEW call library node as an array of 16 U8s or a cluster of two DBLs and get a successful operation.  For generic operation, you can use an array of U8s of the proper size, then typecast to the correct datatype once fetched.

The user still needs to know what is being fetched.  You could write a function to walk the HDF5 dataspace and return a tree control populated with the datatypes of the elements.  Or you could script a control directly (a lot more user friendly, but more work for you).  Or you could rely on the user to look at the HDF5 file and determine the datatype herself.  Unfortunately, HDF5 can handle a more diverse set of data than LabVIEW (e.g. the aforementioned 128-bit integer), so you will need to take that into account.  Fortunately, these sorts of data types are uncommon.  You may want to consider a JKI right click framework plugin

In the end, the user needs a wire containing the datatype to convert the pointer to data returned by HDF5 to real data so they can use it.  I dodged the whole issue and simply copied one of my primitive read or write calls and changed the datatypes whenever I needed something custom.  Until LabVIEW supports true generic programming, that is as good a solution as anything else, and it will be the highest performance.  Unfortunately, it leads to a pair of read/write calls for every datatype you want to support.

[Jason_S]

Hi DFGray,

Thanks for the response.

@> DFGray wrote:

> I can't change the National Instruments policy about publishing variant details (and the internal format has changed a couple of > times in the last few years) but I can offer you a possible solution.

Of course, as Rolf pointed out, we're not really interested in the internal format. Just the interface. In fact, I can go so far as to say just in two interface functions: Variant -> TD + Data Handle, and TD + Data Handle -> Variant.

> Since LabVIEW is a strictly typed language, you really cannot do anything with data unless you know what it is.

Not so. Let me present my common use case. Since you've dealt with HDF5, you should be familiar with attributes, which are (like attributes everywhere) simple name-value pairs associated with another object, e.g. a dataset. When I'm actually using my HDF5 library (and not writing it) I commonly store all sorts of stuff in attributes, e.g. data and time or experiment, type of instrument, comments, phase of moon, favorite musician at the time, etc. When reviewing the data, it's nice to see all of those attributes, and a great way of doing that is to read them all back into two arrays: a string array of the names, and a variant array of the values. From there, I use the OpenG Format Variant to String to convert all of the variants into strings and display all the attributes in a table. Since the attributes all have different data types, an array of variants is the best, easiest, and most general way to accomplish this. This may be single use case, but, at least for me, it's a common case.

And before someone says "store them all as strings," what if I want to also use a specific attribute as a number? What I didn't write the file in the first place?

 

> In the end, the user needs a wire containing the datatype to convert the pointer to data returned by HDF5 to real data so they

> can use it.  I dodged the whole issue and simply copied one of my primitive read or write calls and changed the datatypes

> whenever I needed something custom.  Until LabVIEW supports true generic programming, that is as good a solution as

> anything else, and it will be the highest performance.  Unfortunately, it leads to a pair of read/write calls for every datatype you

> want to support.

Yes, and I intend to support every data type that LabVIEW supports. Well, that's not quire true. I won't put any effort into the useless signal datatype. There may be a few others as well. Ultimately, the problem is not the basic data types, it's clusters and the unlimited types that arise because of them. In order for the interface to be worthwhile, you must be able to handle clusters, and those cluster must be generic. Since the clusters themselves may have non-flat elements in them (notably strings) it quickly becomes problematic to handle them in any non-programmatic, strictly-typed way.

As to HDF5 handling a more diverse data set, this actually isn't a huge problem. In many cases, it can automagically convert between types on the fly. And, regardless, you're correct about using the same sized buffers if necessary.

I've been thinking about this library for about six years now, Some scripting developments have made it possible to really do the library correctly without primatives. If I could solve the variant issue cleanly, I'd really have a snappy product. Well, that, and if could figure out why I'm currently causing memory violations :). Meh, bugs. It's what development is all about.

Jason

[DFGray]

I see your point, but have a couple of workarounds for the case you mention.  I also realized a further incompatibility between HDF5 and LabVIEW.  First the incompatibility - LabVIEW clusters can contain any sort of information, including arrays and strings.  HDF5 compound datatypes are of fixed size, so are a poor way to realize LabVIEW clusters - a generic solution would use groups, attributes and datasets to recreate the cluster.  Arrays of clusters would need indexed groups or something similar.  The reverse is not true.  LabVIEW clusters work great for storing HDF5 compound datatypes in memory.

For your common use case, you are interested in converting the data to text, not variant.  Variants do give you a simple way to do this, but not the only way.  It would be straightforward, but tedious, to create a serialization function for each datatype you want to support.  I don't think this would be any worse than trying to convert it to a type descriptor and variant, and is supported a lot better.  You could support compound datatypes using the U8 array trick I mentioned above.  I realize you would need to redo this effort for every common LabVIEW datatype you wanted to support, but can't think of another one off the top of my head you would need.

Alternately, you could use h5dump from the command line to get an XML version of your data and parse that to get your name/value pairs.  This would probably be the easiest way to do it.