Make Class Mutation History Optional

[crossrulz]

This topic keeps coming up randomly.  A LabVIEW class keeps a mutation history so that it can load older versions of the class.  But how often does this actually need to be done?  I have never needed it.  Many others I have talked with have never needed it.  But it often causes problems as projects and histories get large.  For reference: Slow Editor Performance with Large LabVIEW Projects Containing Many Classes.  The history is also just added bloat to your files (lvclass and any built files that use the lvclass) for something most of us do not need and sometimes causes build errors.

My proposal is to add an option to the class properties to keep the mutation history.  It can be enabled by default to keep the current behavior.  But allowing me to uncheck this option and then never have to worry about clearing the history again would be well worth it.

[AristosQueue]

Amusing: the last time I talked to LV R&D on this topic, it was about adding the mutation history to regular typedefs. 🙂 That conversation even included: "You know, if every VI was just saved as a list of edits made to it, instead of just saving the final panel/diagram, then we could almost completely solve the source code control merge problem by merging instruction lists." That was an interesting thing to consider.

But I do understand why sometimes the mutation history is a problem, so let's discuss cutting it.

This is a quick data dump that Christina asked me to write up a few minutes ago. I reserve the right to come back later and say "I forgot about XYZ..." It has been a few years since I've worked on this. 🙂

crossrulz said in the Idea post:

"But how often does this actually need to be done? I have never needed it. "

The class mutation history affects more than you think.

1. Flatten/unflatten from string between versions

2. Flatten/unflatten from XML between versions

3. Updating default values of controls/indicators/constants

4. Updating Bundle and Unbundle nodes

It's #4 that people find surprising. Typedefs of clusters have a number of behaviors that are frustrating to users that have to do with multiple updates to the typedef over time. If you edit a cluster typedef with its callers in memory, those callers get updated, including their bundle and unbundle nodes and values. And if it isn't a reproducible edit, the caller VI gets a docmod (even when using source only). As long as you a) have all the callers in memory when you edit and b) save everything that LV tells you needs saving, you're fine. But if either of those isn't true, you can end up in a situation where the edits of A->B->C are not the same as A->C, and callers that were in memory during edit B change in a different way than ones that weren't in memory.

Classes have the mutation records to help this problem. Every rename, reorder, and replace is known, so the bundle/unbundle nodes can be updated accordingly.

Now, caveat here: since bundle and unbundle nodes of classes can only appear within the members of the class, and because a class loads all of its members into memory, it is exceedingly rare for an (un)bundler of the private data control to not be in memory when an edit gets made, and most users will save all the callers. But the mutation history is there to backstop you.

Do We Really Need It?

Use case #1 is uncommon. #2 is more uncommon. #3 is almost unheard of. #4 is very rare. Given how rare it is, it's pretty easy to say, "Let's kill the mutation history." 

So why does it exist? Why did I spend almost a year on it and NI bother to patent the algorithm behind the mutation if it is such an edge case?

The simple answer is: you don't realize you need it until you need it, and if you haven't prepared for it, you're up against one of the hardest problems in information communication. By including the mutation history in every class, I warded off entire categories of problems that users could encounter with data changing over time, problems that literally fill textbooks. The amount of code that devs need to handle data mutations in most languages is huge. In LabVIEW, it's automatic. And I've known many customers who have used the features quite successfully.

Histories Grow. Why Is That Not A More Common Problem?

Mutation histories are generally small. The instructions for how to handle renames and reorders are very simple, and I store them in a compressed notation. You can make thousands of edits without getting particularly large. But there's one edit that causes bloat: default values of fields. As soon as you start setting default values of fields, now the mutation history grows as large as the data therein. Put a giant array or a picture control or a string? You'll have a copy of that moving through your system.

Most histories are small... most classes don't set default values for fields. Therefore, mostly the history can sit there ignored and be a life saver for the few users that stray into needing it. So, I decided to include it.

Could LV Do Something Besides Wipe Out Mutation History?

I said above that data values get copied off at every mutation. I didn't do anything to single-source those values. That means you can end up with the same array copied multiple times. LabVIEW could change the mutation records to only store one copy of any given data value and just point at it. That would reduce the overhead considerably. But it would still be a problem for that one time that a user included a giant array in the value -- even after deleting it, it's a part of the class history.

So, Make It Optional? Mostly Yes.

I'd support making it optional, but I'd have it on by default and let people turn it off. But there's a catch here: if any ancestor class turns it off, it has to be off for all descendants. Likewise, if an ancestor is designed to work with it, every descendant needs to have it. So I would make it settable if and only if the class' parent is LabVIEW Object. (There are existing settings that are only available on the root ancestor class, so this isn't new.)

What About Format of Flat String?

If we eliminate the mutation history from a class, we could in theory make the flattened form of the class smaller by removing all of the version information. Should LabVIEW do that?

I'm thinking maybe that needs to be a second option on the class -- if you turn off mutation records, then there's a second checkbox for "use smaller flattened format?" The reason to make it an option is to allow people to turn the mutation records back on later, and, viola, the flattened strings would already have the mutation version in them. But making it a checkbox would allow a much smaller string in some cases, which may be valuable to some users facing bandwidth or file constraints. OTOH, that would complicate the parsing of flattened classes for people who read the strings outside of LabVIEW.

I don't know what I'd do there -- needs more thought.

What Happens If Mutation History Is Missing?

No big deal. LabVIEW is already fully tested and resilient against a class not having a mutation history. Reach into your classes and delete that section? All good. You'll get some errors trying to unflatten old versions, but those are expected errors. And you'll get some load warnings about default values reverting to default defaults -- same as you get on typedefs. Basically, LabVIEW is already ready for this change... it just needs a UI that is more than someone opening the .lvclass in a text file and deleting some XML.

[AristosQueue]

> "Any edit that changes the data type of the private data cluster bumps the version number of Alpha." Any edit?

> So when I add, delete, add, delete something, the version increased by 4? I don't think so.

Until the class is saved, it's not an edit. And then it gets bumped.

[AristosQueue]

> "If you add an element to the cluster"... What if I replace the entire cluster with a copy that has an extra element?

> Will that be managed?

That's a replace. It will be handled by coercion, just like non-clusters. How does a cluster of one type coerce to a cluster of another type with extra elements? It doesn't.\

> "If you reorder elements in the cluster, " Reorder how?

Reorder is a defined operation. Literally, reorder. The other edits are discussed separately.

> Reordering a class's private data has never not crashed LabVIEW for me, so I learned not to use it.

I use it a lot (did earlier today, in fact). Works for me, and I don't recall ever getting a bug report on it. If you have such a crash, send it to NI. But when I used the term "reorder", I meant reorder the menu item, not any other edit.

EDIT: I was originally thinking of dragging out to another panel and back in... I wasn't thinking of dragging within the private data control. As long as you stay within the bounds of the PDC, that should also be a reorder. The editor tracks the movement of the specific controls as they move in the PDC. Sorry for that confusion.

[AristosQueue]

PS: Somewhere on LAVA is a much deeper dive into the mutation history. And there are VIs in vi.lib to let you inspect it and modify it.

See this directory on Mac (Windows users, I trust you can find your way):

/Applications/National Instruments/LabVIEW 2020 64-bit/vi.lib/Utility/EditLVLibs/LVClass

[avogadro5]

The automatic mutation is, to me, a major encapsulation break. When I look at a class, I expect to be able to ~perfectly understand how the private data can change by looking through the methods and their scopes (with the assistance of the very useful "find all private data uses" IDE plugin). With the addition of mutation histories and serialization, now I have to understand the whole mutation history which has not even been made visible by the IDE.

When I make changes to a "mutable" class I also need to be able to test the mutations but opening multiple versions of a class at once is not readily done in LabVIEW which makes that a headache...

[AristosQueue]

avogadro5: deserialization is the encapsulation break, not mutation. If you're concerned about encapsulation breaking (and it is a reasonable concern), we need to disable the ability to unflatten from string entirely -- something I STRONGLY considered doing early on, making it mandatory for every class to opt into deserialization. If I can flatten the class to a string and then unflatten it, I can change the contents of that string before it goes to unflatten. If there's no semantic analysis of the unflattening, there's no guarantee that the object restored plays by the rules of the class.

If you accept deserialization, mutation is actually far more controlled rules of the class than anything coming from another source and being manually manipulated into an acceptable string.

If you aren't in control of your string sources, then, yes, you need to have a method on the class to vet the objects post-deserialization for validity. Or else guarantee that all field data values form valid objects.

I surveyed and found that the vast majority of classes (looking at other programming languages back before LVOOP and then double checking years after LVOOP) aren't operating under such threat vectors. Given than very few programmers I spoke with could even comprehend the nature of the problem and even those who could weren't inclined to write validation methods for every deserialization operation, I concluded that the ease of use of having serialization and deserialization always available outweighed the encapsulation concerns.

[AristosQueue]

> When I make changes to a "mutable" class I also need to be able to test the mutations but opening

> multiple versions of a class at once is not readily done in LabVIEW which makes that a headache...

You just need to create the flattened strings once for each version as you create it. Then your tests should continue to pass as you add new versions or else you're going to need to do something custom in your deserialization.

[Thoric]

I don't know how many times the mutation history has been useful, it would be good to get a feel for that before I check a box that turns it off for good.

Maybe I'd prefer the mutation history to remain on, and for there to be a simple way to expunge it from existing classes if I think I need to?

Maybe a right click menu item for lvclass project elements. Or in the Project Properties similar to Separate Compiled Code.

I'd be annoyed if it was a purely on/off option and then I found there were loads of scenarios where it was saving me time and now I can't benefit. 

I guess I'm saying, for me, I'd like to be able to clear that mutation history at specific points when I feel it matters. Such as to compress a class size, or during problem diagnosis (like Clear Cache), or for final releases of software where I 'know' I don't need any legacy history data support.

[IlluminatedG]

At this point I'm convinced no one should ever try directly serializing classes with generic mutation handling. (Since AQ knows these packages) we've completely abandoned Blue Serialization and now just use JSONtext with a modified version of the general object serializer to enable just a subset of class data to be serialized. We're implementing completely separate mutation functionality that runs as a pre-processing step to deserialization since so often larger scope is needed to handle proper mutations.

Why? Refactoring.

Moved inheritance around? Can't be handled the way you need without additional functionality. Moved data between classes for better dependency management and encapsulation? You're writing your own conversion code again. There are so many situations that breaks simple generic mutation and those are all the situations we ran into as we tried to cleanup code bases.

I'm sure there's some level this needed for core LabVIEW capabilities as stated by AQ, but I can't say I've ever been impressed by LabVIEW's ability to ever update anything that wasn't in memory when a change was made. There's sometimes the promise of intelligent behavior by the editor but I can't claim to ever have laid witness to it.

[fefepeto_kb]

I just realized that nobody mentioned this, but I have insane objects on FP of the class controls sometimes. It is mostly relevant if I have classes that own other classes.

Maybe my structure is deeply nested, and causes the problems, but the only solution seams to be to remove the mutation history.

If someone can confirm this, it would be great, because it might be a bug.

On how to reproduce it, well, I think it is hard, since it has to be a large enough codebase to cause issues, with multiple classes and datatypes around.

Also worth to mention that using interfaces help making the problem less prominent.