Maintain Mutation History as part of Enum Type Definition

[JackDunaway]

This Idea was spawned from my quest for the über-migration experience when upconverting previous versions of enum data to the most recent version - in other words, providing backwards compatibility for data sources (networked clients/servers, databases, files...) that use a previous version of an enum. 

---

When you flatten an enum, the result is the only the numeric value of the name-value pair. Once flattened, an enum is indistinguishable from a vanilla numeric, containing no additional information:

            

(Yes, the snippet works in 2010, even with the cameo, but it's slow because the "Favorite Scrabble Word" enum has 47k+ items)

Looking at the flattened value, the first 4 bytes (00000001) represent the value of the I32, the next two represent the 6809th enum element (1A99), the next two represent the decimal year in hex (0797), and the final two bytes represent the birth month enum index and day of month respectively. That's all the bytes there is - 20 hex nibbles of flattened data to describe the value.

Storing this data structure as-is will be problematic if the type definition of the Scrabble Club Member ever changes. Fortunately, a typedef can be converted into a class, and the class definition will natively retain data structure mutation history, meaning flattened object data can be upconverted from previous versions. Cool!

Above, we can see that if the data structure is converted to a class, the red-underlined represents byte-for-byte the same information as the flattened cluster, but with flattened classes there are additional goodies buried in the data, such as the version of the data underlined in blue.

From Preserving Class Data:

"LabVIEW, as a graphical programming environment, has an advantage over other programming languages. When you edit the class, LabVIEW records the edits that you make. LabVIEW is context aware as you change inheritance, rename classes, and modify private data clusters. This allows LabVIEW to create the mutation routine for you. LabVIEW records the version number of the data as part of the flattened data, so that when LabVIEW unflattens the data, LabVIEW knows how to mutate it into the current version of the class."

But what if you mutate one of the enums, by adding items, removing items, rearranging items, or renaming items? Now you're in a pickle, because Enums currently do not store mutation history as part of their definition.

---

Consider the simple enum, GreekLetters, as part of a project developed by So-So Engineering, Inc. The SSEI developers launch the code into production with version 1 of the enum: v1: { Alfa, Betta, Hero, Delta } (The So-So engineers aren't too good at Greek, but so-so engineering brings home the bacon at SSEI).

Throughout a long development cycle, every few months this enum is mutated to account for mistakes, new features, or obsolete features:

1. v2: { Alpha, Beta, Gyro, Delta }
2. v3: { Alpha, Beta, Gyro, Delta, Gamma }
3. v4: { Alpha, Beta, Delta, Gamma }
4. v5: { Alpha, Beta, Gamma, Delta }

The SSEI developers pounded out code in blissful ignorance until v4 when the app started outputting thingamabobs rather than whirleygigs. 

Their superstar programmer - boasting his shiny CLAD certificate he had been toiling for 3 years to obtain - realizes the previous versions from persistent data logs were being interpreted incorrectly. With the new v4 enum, v1 data with index 2 is now being interpreted as "Delta" instead of "Hero", and v2 data with index 3 is now "Gamma" instead of "Delta".

v5 comes around with a homebrew-upconverting-state-machine-mutater, and continually ranks higher as nastiest code the guys at SSEI have ever seen as they slowly discover Epsilon in v6, Zeta in v7, Eta in v8...

I propose that this revision history, or mutation history, be stored with the control type definition of the enum itself. Further, a flattened enum should include a four-digit version (like classes) in addition to the flattened value.

This Idea may be manifested as a new data construct altogether - the 'Enhanced Enum' or the 'Enum with Benefits' 😉 - or more elegantly it could be included as a new flag 'IncludeVersionHistory' in the existing enum construct.

As a final note, even if you think you're immune to this problem since you don't write enums to persistent files to be read by future versions, you're probably affected. A subVI that uses an enum BD constant will improperly upconvert to the most recent version if it is not in memory while the enum goes through multiple mutations - say a rearrange and rename - that cannot be implicitly determined without explicit mutation history.

If that explanation does not make sense (hey, it doesn't even make sense to me), just ask yourself this question: "Do default values of controls and BD constants sometimes get screwed up after rearranging, adding, or renaming enums?"

Finally, while you're excited about the Enhanced Enum, also get excited about the Enhanced Enum supporting sparseness and different datatypes. Feel free to post questions/comments/insults in the comments section below or on the original thread.

[AristosQueue (NI)]

When data mutation for LV classes was developed, we talked about doing it for typedefs too. Since typedefs flatten as raw data, there's no way to know what version of the typedef was flattened, and that has to be part of the flattened string to do the unflatten-and-mutate. As you point out above, it is not enough to store the mutation records in the typedef. We'd have to change the flattened form of the output as well. That was considered a showstopper unless we developed a brand new kind of typedef that was a "Versioned Typedef". The amount of work was estimated to be very high.

Further, it isn't clear that something like an Enum would actually benefit. For example, in the cluster editing, doing a Replace of a control creates a different mutation record than deleting a control and adding a control. When users are working in the clusters, they typically choose the operation that they are thinking about. With Enums, they just edit the text until it looks like what they intend it to look like, making it ambiguous about where all the values should end up. Also, if they delete the last element, when the value that is unflattened from disk is the old last element, what value should it become? With a deleted control, we just throw the data away. But we have to give *some* value for the enum. That means that for enums, we'd probably have to provide an even more sophisticated UI for specifying how old values should be mapped, rather than just learning it from the user's edits the way we do with the private data cluster. That ups the amount of work further.

None of this means that the idea is a bad idea. It just means that there's a lot that would have to be worked out to make the feature truly viable.

[JackDunaway]

>> When data mutation for LV classes was developed, we talked about doing it for typedefs too.

Even though the concept of storing mutation history in the type definition applies to typedefs or classes or any data structure in general (it is not specific to any OOP tenet, it just happens to be implemened in LVOOP), I'm only concerned with enums since that's where there's a hole in the language. If you want to store the mutation history of a typedef, just turn it into a class... not too big of a deal.

>> Since typedefs flatten as raw data, there's no way to know what version of the typedef was flattened, and that has to be part of the flattened string to do the unflatten-and-mutate. As you point out above, it is not enough to store the mutation records in the typedef. We'd have to change the flattened form of the output as well. That was considered a showstopper unless we developed a brand new kind of typedef that was a "Versioned Typedef".

Probably a better name than "Enum with Benefits" 😉 . Did the design committee consider having an additional flag on the enum type definition called "IncludeVersionHistory", which would trigger the Unflatten/Flatten prims to behave differently?

>> The amount of work was estimated to be very high.

Better you than me! In all seriousness, hard-hitters could save some serious time not needing to worry about app-scoped mutation mitigation. Virtually all of the mutation code I have written could have been captured by a macro, and I would not be surprised if most people's mutation code is also "monkey work". It boils down to the benefit-vs-cost white collars, but all I can do is herald the end-user's benefit in hopes of justifying your development/testing/maintenance/documentation cost.

>> That means that for enums, we'd probably have to provide an even more sophisticated UI for specifying how old values should be mapped

I disagree - the current enum editor is sufficient for fulfilling my request. (albeit, this is where the obligatory plug for a Better Enum Editor comes in). Any developer relying upon inherent the mutation capability I'm proposing needs to know the difference between hijacking an obsolete element and renaming it versus deleting the obsolete element, adding a new one, and rearranging the new element to occupy the obsoleted element's slot. You would expect these two mutations should provide two different outcomes, both of which are necessary for different situations.

>> Also, if they delete the last element, when the value that is unflattened from disk is the old last element, what value should it become?

For my proposed inherent mutation, any obsoleted values would become the default value of the enum. To have the value react in any other way would require an app-controlled mutation function, adopting the same philosophy for custom-vs-inherent migration of versioned classes.

[Intaris]

I always save Enums as strings.

I have trouble envisaging locations where I would run into this problem.

[Manzolli]

Looks like it's something that people don't need often, but in the way it is now implemented could lead to serious problems!

[JackDunaway]

>> I always save Enums as strings

Yeah, me too. But consider this conversation:

Developer #1: "I don't need enums to version properly - because I save enums as strings."

Developer #2: "You save enums as strings? Why don't you save enums as enums?"

Developer #1: "Because enums don't version properly"

Also, keep in mind, saving enums as strings is not resilient against name changes, whereas enum mutation history can handle all mutations (rename, rearrange, add, delete).

[Intaris]

I was never personally aware of a versioning proiblem with enums (I never encountered it).  As such I just do it because when I see my saved data I can immediately recognise what I just saved.

[tst]

> when I see my saved data I can immediately recognise what I just saved.

You can, but can your program?

Suppose I have a configuration cluster which I save to the disk and reload later. Suppose further that this cluster has an enum with an element called "spectral shift". Then suppose that I present the application to The Big Manager of The Big Client who points out I forgot the F in "shift". The next natural supposition would be me fixing the enum. It's a typedef, so the fix is easy.

The next part, however, would be a client running the app and not understanding why the value they saved wasn't being loaded. Not very nice, is it?

[JackDunaway]

>> As such I just do it because when I see my saved data I can immediately recognise what I just saved.

My perspective is this actually an undesirable trait of a binary file. If you want human-readable, go whole hog XML. Binary is mean for compactness and speed (no parsing).

>> You can, but can your program?

Currently, not without some homebrew wackass mutation code 😉

[Intaris]

I nearly always save all my data in human-readable form.

Regarding reading in old values..... I just have never come accross this problem, honest.  Maybe I'm just lucky.

[Daklu]

"Maybe I'm just lucky."

You won't run into enum mutation problems with certain kinds of edits.  Changing an element's spelling or adding elements is okay.  Reordering or deleting elements is bad mojo, unless *every* vi that uses the enum is loaded in memory.  Once your enum is exposed outside of the module that defines it, via fp terminals, the ctl itself, or by being saved on disk, you cannot reorder or delete elements without risking broken code.

I almost never expose enums as part of a component's public interface for the very reason that is almost impossible to change them safely.

------------------------------------

"When users are working in the clusters, they typically choose the operation that they are thinking about."

I must be atypical.  I choose the operation that will get the job done quicker.  It's usually quicker to delete the old element and drop in a new element than navigate the context menus, palettes, and file structure to get the new element.

------------------------------------

With all due respect, I don't like this idea.  The problem with enums isn't that they don't store mutation history, it's that many programmers overuse them and are too quick to edit them without understanding the consequences.  The convenience of wiring an enum into a case structure allows programmer to learn bad habits wrt editing enums.

Enums are an aid for programmers reading source code.  They allow us to assign human readable names to magic number values, but the number, not the name, is what's important.  The fact that they are numbers makes them fast and small.  Carrying around the mutation history turns an small data type into a relatively gigantic blob of data.

Automatic mutation has its drawbacks as well.  For starters, it's an opaque process that imposes an arcane set of rules on how the data type is edited, not on what the result of the edit is.  If you do it wrong you've hooped backwards compatibility, possibly without even knowing that you've done it.  Another issue is that there's no way to inject your own mutation code in those cases where auto mutation isn't working or can't be tested.

IMO "sticky" rings (the ring-based equivalent of sparse enums--the label 'sticks' to the value) is a more appropriate solution to the problem of wanting to change the order or content of the enum after it has been released publicly.  Yes, you will still have to write mutation code; I think it would be easier to do though.  Yes, the newly minted CLAD at SSEI will still screw it up.  When he learns how to do it right he can submit a resume to Excellence In Engineering Inc.

That's my $.02