Avoiding 'vip hell' with VIPM packages

[vishots.com]

Some suggestions:

• Less preffered method: Include the old version of the VIs in the new package so old linkages are not broken. Same package, but now it's slightly bigger since it will contain the older VIs.
• Recommended method: Package the old VIs into a new package that is a dependency of the existing package. When the user installs the new package version, all is good, since the dependency will also be installed which supports the old code. This allows you to sandbox the old code and you can also provide bugfixes to the old code without forcing an upgrade of the exisitng main package. An additional benefit is that it allows a user to install the main package and not installing the dependency if they don't need it.

[AristosQueue (NI)]

Has anyone ever explored allowing partner program users to hook into LV's version mutation system? We have a mechanism that allows our toolkits to rewrite a user's block diagram during load to retrofit an older VI to use a new API. There are actually two such mechanisms, one tied to the LV version number and the other tied to the version number of our toolkits. The latter is written in G to the best of my knowledge and seems like it would be extensible.

You now know as much about it as I do -- I've never used it because my mutations have always been introduced with new versions of LabVIEW, never in an independent toolkit. Some of you might contact the Partners Program folks and ask them to dig into it.

[Daklu]

David,

Thanks for the response.  Should I read it to mean NI does *not* want me to set up my packages for sxs installation?  (I think sxs is the better solution since it gives the end user more control over upgrades, but I'll do it however NI says it should be done.)

-------------------------------

Michael,

Let me try to describe what you're saying to see if I'm understanding your recommended method correctly...

Currently all my package names have a version number appended to them and each major version is installed in its own directory.  You are suggesting not appending a version number to the current major version, but continuing to install it in a directory with a version number.  For example, instead of doing this,

v2 is current

Package Name:  LDM.v2.vip

Library Name:  LDM.v2.lvlib

Install Directory:  <vi.lib>\LapDog\LDM.v2\

I'd do this,

v2 is current

Package Name: LDM.vip          <--- No version number on package name

Library Name: LDM.v2.lvlib

Install Directory: <vi.lib>\LapDog\LDM.v2\

And when a new major version is released I'd do this,

v3 is released

Package Name: LDM.vip         <--- New version has same name as old version

Library Name: LDM.v3.lvlib

Install Directory: <vi.lib>\LapDog\LDM.v3\

Package Name: LDM.v2.vip        <--- Prior major version gets a new package with a version suffix

Library Name: LDM.v2.lvlib

Install Directory: <vi.lib>\LapDog\LDM.v2\

Did I capture your idea accurately, or are you suggesting the current major version always use the same namespace/directory and move prior major releases to a new namespace/install directory, like this,

v2 is current

Package Name: LDM.vip

Library Name: LDM.lvlib

Install Directory: <vi.lib>\LapDog\LDM\

v3 is released

Package Name: LDM.vip           <--- This package is now version 3

Library Name: LDM.lvlib             <--- Replaces v2 namespace

Install Directory: <vi.lib>\LapDog\LDM\            <--- Replaces v2 on disk

Package Name: LDM.v2.vip      <--- Prior major version gets a new package with a version suffix

Library Name: LDM.v2.lvlib        <--- v2 namespace changed

Install Directory: <vi.lib>\LapDog\LDM.v2\        <--- v2 install location changed

[David_L]

Hi Daklu,

The ideal solution would be as the developer to handle major changes nicely so that different versions are compatible and side-by-side installation is not needed.  Therefore any packages depending on your toolkit should not care which version is installed and everyone plays together nicely. 

In an extreme case where the change is significant enough where this is not possible, then you would want to rename the package and namespace so that they can both be installed together.  This is similar to the situation of DAQmx and Traditional DAQ.  You would then deprecate the old version and tell new developers to only use the new version.  However you should try to avoid this situation if possible.

Hope this helps,

David

[David_L]

AQ, I didn't know we had something like this in G, but I like the possibilities. I'll look into it more to see if it's easy enough to expose this to partners...

Thanks!

[Jim_Kring]

Yes!  This would really help out add-on developers and improve the API deprecation and upgrade process.

[David_Staab]

I love it when the heavy hitters start posting to a thread together! I learn so much so quickly!

I'm familiar with WinSxS as a solution to DLL hell, and very likely as Daklu's inspiration for choosing the nomenclature and in his original post. As a mostly academic student of versioning distributed software components -- the few I created while working for NI haven't been majorly revved for public release, so I had control over my users' upgrade decisions -- I like that approach more than others that attempt to "gloss" over the incompatibility between major revisions. The only real disadvantage I've read about so far is that it has a tendency to use lots of disk space if you depend on several versions of each package.

For power developers, it affords a lot of flexibility, though: we can decide which version of a distribution to depend on based on our component's needs, our availability to upgrade dependencies, and the business risk associated with performing an upgrade. Coming from an automated test background, I'm used to the "never change the deployed system's dependencies" mentality (a variant of "if it ain't broke, don't fix it").

However, David L. has decreed that SxS is to be used only as a last resort. To that end, I guess Michael A.'s recommendation is the way we have to do it?

MichaelAivaliotis wrote:

Recommended method: Package the old VIs into a new package that is a dependency of the existing package. When the user installs the new package version, all is good, since the dependency will also be installed which supports the old code. This allows you to sandbox the old code and you can also provide bugfixes to the old code without forcing an upgrade of the exisitng main package. An additional benefit is that it allows a user to install the main package and not installing the dependency if they don't need it.

If that's the case, then I'm very interested in hearing an answer to Daklu's question of whether to keep the namespace in a newer version (as well as the file name), or to use a new namespace for the new rev. This might be a good thing to discuss in the next post on the LVTN Dev Center blog: having a step-by-step guide to referene would be very handy.

[Daklu]

DavidStaab wrote:

I'm familiar with WinSxS as a solution to DLL hell, and very likely as Daklu's inspiration for choosing the nomenclature and in his original post.

Yep. I'm not shy about coining new terms when I think there are notable differences, but in this case WinSxS seems like a reasonable model to follow. It's not perfect, but it's better than ignoring the issue and letting the problem creep up on us.

DavidStaab wrote:

For power developers, it affords a lot of flexibility

Exactly. As an api developer I can't predict all the situations users will find themselves in regards to which packages will need to be loaded. SxS installation appears to be the best option available.

DavidStaab wrote:

However, David L. has decreed that SxS is to be used only as a last resort.

I interpret his comment as meaning, "If you are breaking backwards compatibility go ahead and do sxs installations, but please think long and hard before making any breaking changes." Releasing 3 different major versions within six months is a sign I'm being too loose with my api's interfaces. On the other hand, as I described earlier an api *has* to evolve and be allowed to break backwards compatibility during its life cycle. Otherwise I have a chicken and egg problem: I can't release an api without knowing how users will want to use it, and I can't know how they will want to use it without releasing it.

Speaking for myself I would expect v1 and perhaps v2 to have relatively short life cycles--perhaps a year or so each. As I learn what users are doing with it I can make changes to simplify the programming model and make the api easier to understand and use. By v3 I would expect more long term stability with breaking changes only occurring when NI introduces applicable disruptive technologies. (i.e. Interfaces, etc.)

DavidStaab wrote:

If that's the case, then I'm very interested in hearing an answer to Daklu's question of whether to keep the namespace in a newer version (as well as the file name), or to use a new namespace for the new rev.

Me too. I hope Michael comes back and comments on it. I've been thinking about the options I described and the more I think about them the less I like them. It's not clear to me what use case they are supposed to address. (Maybe I didn't accurately describe what Michael had in mind...?)

Option 1 - Current package is unversioned, current and prior library names have version info

To be honest this doesn't make sense to me. Installing package v3 removes the v2 code. Since they have different namespaces and different install locations any applications using v2 are going to end up with missing vis. The workaround is to list the v2 package as a dependency of the v3 package so the v2 code will be reinstalled when the v3 package is installed. Of course, v3 isn't really dependent on v2. We're just saying it is so the user's code doesn't break.

Option 2 - Current package and library name is unversioned, prior library names are versioned

This makes less sense to me than option 1. It has all the problems of option 1 with more issues on top of it. The whole point of rolling a major version number is to indicate compatibility has been broken. Replacing v2 with an incompatible v3 library in the same directory and with the same namespace would be disastrous.

The common thread is using an unversioned package name as the current release. (I'll ignore option 2 since it creates more problems than option 1 and doesn't appear to be a good solution.) I'm having a really hard time seeing the upside to this idea. Maybe it's supposed to simplify things for end users so they don't have to deal with version names? How does it affect user scenarios?

• VIPM lists three packages: LDM, LDM.v1, and LDM.v2. How will users know which version is the most recent? Obviously v2 is more recent than v1, but it's not at all intuitive that LDM is the current package. Unversioned names could be the current release by convention, but that's still a non-intuitive barrier for new users.

• Suppose someone builds code against v1 when it is the current release. How do they document it's use? It's listed as "LDM" at the time, but at some point in the future it will be listed as "LDM.v1." Do we require users to update their documentation when we release a major revision?
• How is a user supposed to know the difference between a "real" dependency and a "fake" dependency specified to prevent breaking existing code?
• Installing the current package also installs all previous versions by default. Taking it to the extreme, if v9 is current versions 1-8 are also installed. That's a lot of extra baggage handling "just in case" the user has code based on prior versions.

I dunno... I can't see how an unversioned package name benefits the end user. It just seems to add questions and confusion where none is needed. I find it much more straightforward and clear if each major version is distributed and labelled as an independent package. If I need v1, I install v1. If I need v9, I install v9. Each is clearly named and there's no confusion.

In my mind packages and namespaces have a very close relationship.  (Which explains why my package names reflect the namespace of the contained libraries.)  Having a single package remove one namespace and install a different one is very uninuitive behavior.  I'm open to being pursuaded by alternate solutions, but I need to understand how it is easier/more flexible for end users than sxs packages with version info in the names.

[vishots.com]

Why can't you just create a package with a new version and just keep the old compatibility VIs in the package? Wouldn't that support SxS? Are you concerned about large packages?

[Daklu]

@Michael - You're referring to the 'less preferred' method you described?

We might be miscommunicating on the scope of changes I'm talking about here.  For small changes like changing a vi's connector pane I'd create a wrapper vi, deprecate the original vi, and release it as a minor update.  SxS installation is reserved for big changes that break compatibility, like changing an inheritance hierarchy, moving classes between libraries, etc.  In my experience these kinds of changes are very difficult--and often impossible--to do "in place" while preserving backwards compatibility.

Maintaining compatibility in these situations isn't a matter of just keeping the prior vis in the package.  I'm replacing entire families of classes and libraries.  Namespaces are changing.  In principle I could put all the new code in the same package as the old code, but in practice there's no benefit in doing so and (IMO) valid reasons not to package them together.

1. Most, if not all, of the classes in the old version will be replaced by similar classes with a different namespace in the new version.  All the member vis on the user's block diagrams will need to be replaced too, since they have a new namespace as well.  What this means is users wanting to upgrade their code from the old version to the new version will need to do about the same amount of work regardless of whether the new version is in the same package or in a new package. 
   
   i.e. Packaging them together doesn't gain us anything.
2. If I were to have every major version of the package also install all prior major versions then I have conflicting use cases.  I need to install the palettes for each major version because even though a user might have installed v3, it does not mean they no longer need to use v2 for some of their applications.  They need the v2 palettes.  On the other hand, when a new user installs v3 they should only see the palette for v3 since that is the current version. 
   
   Giving the new user palettes for all the versions raises additional questions in their mind.  What are the differences between versions?  Why are they all included in the package?  Which one should I use for my project?  It also raises questions in my mind.  Will they know objects from v1, v2, and v3 are not cross compatible, even though the classes and vis have similar names?  Will they know why they get a broken wire when they accidentally wire a v2 Message object into a v3 MessageQueue object?
   
   i.e. Packaging them together forces me to compromise the usability of one group of users in favor of another.
3. If I were to have every major version of the package also install all prior major versions then when the yearly Labview.Next is released I need to make sure every major package version works with the latest Labview release.  It becomes impossible to retire code even though a particular package version is no longer supported.  What do I do when LDM v1 (a branch I stopped developing over a year ago) doesn't upgrade cleanly to LV2015? 
   
   Choosing not to support LV2015 isn't realistic and I can't very well let it install broken code.  Should I remove LDM v1 from the package distribution?  Now I'm in a situation where LDM v3 (pre-LV2015) installs v1-v3, but LDM v4 (post-LV2015) installs v2-v4.  We're right back in the problem I was trying to avoid, with a user unable to have both v1 and v4 installed at the same time because they use the same package, even though they both may work fine with LV2013.
   
   i.e. Packaging them together increases the maintenance load on package developers, and may lead to unresolvable conflicts.

I admit being surprised at the (perceived) reluctance to adopt sxs installation for VI packages.  We have history to show us what's in store if we ignore the issue.  (Assuming the LVTN continues to grow.)  We have the benefit of seeing how MS solved the problem to show us how to avoid the heartburn and lost productivity dll hell caused.  I honestly thought it was a slam dunk decision.

Obviously I'm missing out on a benefit to unversioned packages that is apparent to others, but I have no idea what it might be.  Hints?