What is the "catch" in writing Variants to a Binary file?

[Bob_Schor]

I'm saving "event" data associated with an experiment that LabVIEW is "controlling".  The data includes such things as the State of several State Machines, Messages to Queued Message Handlers, digital data from hardware (such as Button Pushes), etc.  My "Event Type" is a cluster with three components -- Event Time (an I32 representing ticks of a 1KHz clock), Event ID (an Enum with values such as State, Message, Digital In), and Event Data, a Variant that is the State Enum or Message String or Digital U8 port value.

I wrote a test routine in which I generated some known data of Event Type, wrote it to disk using Binary Writes, then read it back in to an Array of Events doing a "Read to End-of-File" Binary Read.

Somewhat to my surprise, the array of Events looked exactly correct, including the Variant data.

What is the "catch" in storing data this way?  I can see one obvious problem -- if I change the Event ID Enum, I could mis-identify the Events.  Are there things I need to worry about in handling the Variant data?

[I'm now thinking about hare-brained schemes to save the Event ID data in the file, itself, perhaps as a header of some sort.  And before you ask, I'm not quite ready to think about using TDMS ...]

[DavidMaidman]

the main considerations would be that you are able to parse the data when you load that (but you have already pointed that out) but also there is file size overhead to consider as the variant datatype contains information about the data as well as the data itself

[garrettmarsh]

---

@Bob_Schor wrote:

...And before you ask, I'm not quite ready to think about using TDMS ...]

---

Out of curiosity, why not?

A major catch, as you already hinted, is that the Read fails if anything about the variant data structure changes.

[crossrulz]

The only real catch I can think of for the Variant data is if NI decides to change how to save the data type.  I remember people running into problems saving and loading variant data saved to a file when working with multiple LabVIEW versions.  I think it was concluded that NI changed the binary representation of the data type.

As far as your clusters, if you change them at all you will not be able to recover old files.

The enum I will have to play with.  I think those are just saved as a number based on the representation you give it (a U16 will be 2 bytes).  So as long as you don't change the represenation, I don't think you will have problems with the enums as long as you only add items to the end.  Deleting and/or adding in the middle could mess things up.

[billko]

---

@crossrulz wrote:

<snip>

The enum I will have to play with...  

<mini-snip>

Deleting and/or adding in the middle could mess things up.

---

Oh, yeah, I got a little burned with that one.  Good thing it was just something I was pracicing on.  Because changing stuff in the middle actually changes the values that the text is associated with, it can throw off all the the selections throughout your project!  I was dismayed to find all the typedef'd enum constants were changed to reflect the new text associated with the values.  For instance, if the value for "Init" was "1" and "Exit" was "2" and I added "Test" in between, "Test" would become "2" and "Exit" would move to "3" and all the contstants that had "Exit" selected before now showed "Test" as the selection!

So if you have to add something, add it to the end, no matter how unintuitive it makes the enum.  If you decide to delete something from the middle, don't delete it; replace it with something like <not used> to keep the values aligned with the text.  It may look funky on the enum, but it will save you lots of heartache.

Is there a better way to do this?

[Bob_Schor]

In my experience, my "burn rate" on adding (or removing) Enums "in the middle" is maybe 25%.  Many times it just "works", shifting as appropriate, but trust me, I do check everywhere that Enum is used (thank goodness for Find All Instances").

[billko]

---

@Bob_Schor wrote:

In my experience, my "burn rate" on adding (or removing) Enums "in the middle" is maybe 25%.  Many times it just "works", shifting as appropriate, but trust me, I do check everywhere that Enum is used (thank goodness for Find All Instances").

---

Whew - 25% burn rate is about 25% too much for me.  😉