Avoiding 'vip hell' with VIPM packages

[Daklu]

LabBEAN wrote:

Thinking long term, would this move us in the right direction?

Not completely.  There are actually two separate but related problems we're trying to deal with.  The first is sxs package installation.  We have ways of doing that right now by creating a new package line with every release.  All it needs is developer reeducation.  The second problem is loading packages sxs in a single app instance.  That requires unique namespaces for every package instance.

If I'm understanding your example correctly, it permits sxs installation but does not permit sxs loading because the namespaces are identical.  I'm not sure the scheme is an overall net advantage.  It does allow me to have two projects open with each project using a different version of the same package line, something VIPM currently does not support.  On the other hand, it also really increases the chances for accidentally crosslinking to a different version.  I'm not sure that's a path we want to travel down.

(4) LabVIEW could load the latest detected version of the package line into the palettes by default upon initial launch, but this could be overridden for an individual project if a package user wanted to load a legacy version for that project.

Currently palettes are defined at the Labview application layer, not the Labview project layer.  (At least that's how it appears.)  Allowing customized palettes at the project layer is an interesting idea I hadn't considered.  If simultaneously installed packages with identical namespaces is ultimately where we go that could be part of the solution.  Assuming, of course, it's feasible from a technical perspective.  There are some negative usability scenarios flittering around in my head I'd need to nail down before I would support it.

For example, suppose I create MyVi as part of a project that uses a custom palette to point to 1.0.0.2\mt1_Foo instead of the most recent 2.0.1.9\mt1_Foo?  What happens when I open MyVi outside of a project context?  The vi will still link to 1.0.0.2\mt1_Foo, but the default project context will try to load 2.0.1.9\mt1_Foo for the palette.  Conflict.

-----------

I woke up at 4am this morning and spent 2 hours sitting on my couch just thinking about this.  I feel like I'm getting a better understanding of the problem.

Any solution must address conflicting requirements between valid user needs.

1.  Transparent package upgrades.

2.  Multiple package versions loaded simultaneously.

Need 1 requires identical namespaces.  It does not require an identical install point, though life is easier if it is.  Need 2 requires different namespaces and different install points.  To me the namespace issue is the essence of the problem.  Some of my package users will have need 1 and some will have need 2, but I cannot satisfy them both simultaneously.

Currently VIPM supports 1, but not 2.  At the other end of the spectrum Kosta's suggestion supports 2, but not 1.  The part of me that likes flexibility strongly favors Kosta's suggestion, as it allows us to do things we currently cannot do with the existing package model (i.e. 1 package = 1 api for its entire life) and there is wisdom in the idea that all source code changes should be intentional.  The part that likes simplicity shudders at having to manually update source code to that extent and enjoys the work VIPM does for me.  I'll call Kosta's suggestion the "standard sxs" model, since the installation is standardized across computers.

It may be possible to create a 3rd party tool that unpacks a .vip file and alters the namespaces and install point before writing the package contents to disk.  A tool like that would allow users to take any package instance and customize the installation it so it can load sxs with other versions of the same package.  I have no idea how hard it would be to link the palettes, new names, and new install points so they all play together nicely, and I assume these custom installations wouldn't show up in current versions of VIPM.  (Michael, do you have any thoughts on this idea?)  I'll call this idea the "custom sxs" model to reflect the customized nature of each sxs installation.

Ultimately the best solution is going to be driven by development philosophy and the desired user experience.  In these areas I can ask questions and share my thoughts, but I have precious few answers.  My opinion is the standard sxs model is a better long term solution as it seems to be more robust, provided we have tools to properly support it.  (i.e. Package lines/families in VIPM, better mutation support in LV, etc.)  These tools aren't going to be available for some time, so how do we transition from here to there?

Custom sxs installation may facilitate the transition.  A custom package installer (VIP SxS) would help users like Staab who need sxs installation now.  It does not help with relinking existing source code to the custom installation.  We'd need to develop separate relinking tools for that.  The relinking tools for custom sxs installations can easily be reused for the standard sxs model, so it would be a good place to prototype code and get it used in real world situations.  These tools could be developed right now and provide an immediate solution to the problem without disrupting the current user experience with VIPM or NITN.  When the tools supporting standard sxs become available VIP SxS can be retired.  As an added bonus, if at some point along the way we discover a pure standard sxs model isn't necessary, users still have the tools available to handle any scenario.

Thoughts?  Opinions?

[vishots.com]

Currently VIPM supports 1, but not 2

In VIPM you can already add a prefix or suffix during the package build process to namespace your code. So VIPM does have this feature. Can you elaborate on what you are looking for?

[Daklu]

In VIPM you can already add a prefix or suffix during the package build process to namespace your code. So VIPM does have this feature. Can you elaborate on what you are looking for?

The prefix or suffix has to be established by the package creator.  In order to make the custom sxs model work, the package user needs to have the ability to apply different namespacing when the package is installed.  That's the part VIPM doesn't currently support.

Suppose an end user has two package instances, Foo-1.0.0.1.vip and Foo-1.1.0.2.vip.  By default these packages are mutually exclusive.  There's no straightforward way for the user to have both packages loaded at the same time.  VIP SxS is just an alternate installer that allows a package user to override the install location defined by the package and change the namespacing of the new vis.  That way the user can at least get both versions of the same package loaded into the dev environment.

[LabBEAN]

Daklu wrote:

If I'm understanding your example correctly, it permits sxs installation but does not permit sxs loading because the namespaces are identical.... [It] increases the chances for accidentally crosslinking to a different version.

In my example above, if a unique project library (e.g. mt1_1.0.0.2.lvlib) or PPL owns each mt1_Foo.vi, multiple mt1_Foo.vi's could be loaded into memory simultaneously and there wouldn't be any risks of cross-linking, correct?  If there is a concern that package developers won't create project libraries for their reuse, maybe VIPM should? (At home w/o LV and I can't remember if VIPM does this already.)

To clarify idea (4), I think developers should have the option to show any number of versions of a tool in their palette.  For the case where you have a project where you want to continue only using a legacy version of a VIP, the typical use case would be to only show the legacy version of the tool in the palette for that project.

To further clarify idea (4), new projects would commit tool version(s) to the LVPROJ only when a tool is first detected (i.e. used in project VIs).  I do not envision some sort of "latest" setting where a project always loads the latest version of the tool into the palettes.

Daklu wrote:

There are some negative usability scenarios flittering around in my head I'd need to nail down before I would support it.

For example, suppose I create MyVi as part of a project that uses a custom palette to point to 1.0.0.2\mt1_Foo instead of the most recent 2.0.1.9\mt1_Foo?  What happens when I open MyVi outside of a project context?  The vi will still link to 1.0.0.2\mt1_Foo, but the default project context will try to load 2.0.1.9\mt1_Foo for the palette.  Conflict.

[Edit:  Attempting to remove horizontal scrollbar originally inserted as part of this quote.]

I followed you up until "Conflict."

If you create a new project, assuming you're only showing the latest version of mt1 in the palette when creating new projects, then you would only see 2.0.1.9\mt1_Foo.vi in your palette when opening and editing VIs from the new project.  If you were to open MyVi from this new project, MyVi would still load with 1.0.0.2\mt1_Foo.vi.  I would consider this intended behavior.

If you created a new project and wanted to reuse MyVi from an old project but needed it to now reference the latest version of mt1_Foo.vi, then you could copy MyVi to a new location, rename it, and manually re-link it from 1.0.0.2\mt1_Foo.vi to 2.0.1.9\mt1_Foo.vi.  If there were several other VIs to relink, it could be easier to make a sandbox copy your original development folder, open the sandbox project, and use the tool I described in idea (4) for relinking.

The idea of allowing developers to store VIP versions inside the LVPROJ is to make the palette reflect whatever versions of the tool are being used and to facilitate relinking paths for project VIs when there are sweeping tool version changes.  I haven't used the palette API in a while, but if there's not a palette visible flag there today, NI would need to create one to enable dynamically hiding palettes.

[Daklu]

In my example above, if a unique project library...

Ahh. I didn't understand you intended to release each package with uniquely namespaced libraries. In that case, since you are proposing a standard sxs scheme, then no, cross linking would not be an issue. My comments assumed identical namespaces, so my other responses to your post can be ignored.

Let me try explaining your suggstion again to see if I get it right this time...

Package developers start using the standard sxs model. (i.e. Every release is a new package.) VIPM still manages all package installs and uninstalls. Instead of automatically displaying the palettes of all installed packages, LV allows users to select which package palettes are shown. This is to help avoid palette clutter, but it is not related to the overall functionality of your suggestion. (i.e. None of other functionality depends on which palettes are shown.)

There is a separate Package Linker tool built into LV that allows users to define which packages will be used for each project. The list of selected packages is embedded in the project file. The Package Linker displays all the installed packages from each package line and users select the desired package. If the user deselects a package the project uses and selects a different package in the same package line, LV automatically upgrades all source code to the new package. If a user imports a vi that uses a package that is not currently selected, LV automatically adds that package to the selection list. Is this correct?

My initial concern is it requires LV to manage a complex network of relationships between packages, namespaces, and code mutations. How are those relationships defined? Who is responsible for maintaining them? My second concern is the wisdom of building package awareness into LV. Currently LV doesn't know anything about packages. Intuitively I think that's a good thing. It helps keep the complexity manageable. My third concern is it might be a viable end point, but we still have the problem of getting there from where we are now.

VIPM could allow package users to dynamically select versions within a Family to install SxS or uninstall.

What do you mean by "dynamically" install or uninstall?

If there is a concern that package developers won't create project libraries for their reuse, maybe VIPM should? (At home w/o LV and I can't remember if VIPM does this already.)

No, it doesn't do that currently and I don't think it should. At least not automatically. It would be (in my opinion) one of those features that tries to be helpful but ends up just being annoying and difficult.

--------------

Here's another way to look at the situation as it exists right now, along with (of course) some new terminology.

In a tier 1 conflict, MyCode depends on incompatible package versions. The standard solution to resolving a tier 1 conflict is to change MyCode.vi so it uses the same package version. In theory it works. In practice it is not always feasible. A tier 2 conflict is where MyCode has dependencies that in turn depend on incompatible package versions. There is no standard solution to this problem.

Previously (way back yesterday) I thought the custom sxs model was flexible enough to allow developers to overcome any conflicts they might encounter. For example, many times the conflicts in the diagram above can be resolved by doing a custom installation for PkgA-1.0.0.vip, as shown below. The diagram shows the system state after a user installed PkgA-1.0.0.vip with a customized namespace and install point. It could take some effort to relink the source code to the custom installation, but tools would be developed to help with that.

Today I'm not sure the custom sxs model is sufficient. In particular, what happens if a package's source code is password protected?

I did some playing around with a series of password protected classes. As long as you can isolate the linked libraries in their own app context I'm fairly sure it's possible for end users to manipulate the linkages regardless of password protection. A few brief tests with the LV linker convinced me I don't understand it well enough to claim the custom sxs model is always sufficient to resolve conflicts. Even if the custom sxs model can solve the problem, it looks to be a huge headache for anyone who is forced to go down that path with more than one or two dependencies.

At this point I'm thinking the standard sxs model is the only solution that gives us a workable solution.

[LabBEAN]

Daklu wrote:

Package developers start using the standard sxs model. (i.e. Every release is a new package.) VIPM still manages all package installs and uninstalls. Instead of automatically displaying the palettes of all installed packages, LV allows users to select which package palettes are shown. This is to help avoid palette clutter, but it is not related to the overall functionality of your suggestion. (i.e. None of other functionality depends on which palettes are shown.)

Correct.  If you created a new VI from the Getting Started Window and right clicked the diagram, you would see a package Family palette containing all currently installed Child packages, where each Child package would have it's own sub-palette to the main Family palette.  For projects, we could limit the number of palettes a user sees:  This is to reduce clutter as you say but also to prevent a package user from unkowingly and inadvertently grabbing a subVI from an unsupported package.

Daklu wrote:

There is a separate Package Linker tool built into LV that allows users to define which packages will be used for each project. The list of selected packages is embedded in the project file. The Package Linker displays all the installed packages from each package line and users select the desired package. If the user deselects a package the project uses and selects a different package in the same package line, LV automatically upgrades all source code to the new package. If a user imports a vi that uses a package that is not currently selected, LV automatically adds that package to the selection list. Is this correct?

Not really.  I was trying to address the case where a package user wants to upgrade to the latest package Child in the package Family.  SxS makes this tough for reasons you have explained.  So, I propose a project dialog where a package user could see a list of Child packages that are used in the project for each package Family.  This list could be entitled "From".  Beside each Child package in the list there could be a "To" ring control where he/she could select a different Child.  When the package user presses OK, the linker just changes the subVI paths for affected project VIs.

For example:  If a package user attempts to switch all references of package Child A to package Child B, LabVIEW would just need to inspect and change instances of "Child A" to "Child B" in the dependency paths of affected project VIs:

From:  vi.lib\Package Family\Package Child 1.0.0.1\Foo.vi

To:      vi.lib\Package Family\Package Child 1.1.0.0\Foo.vi

If Foo.vi doesn't exist in package Child B, then the package user will see broken run arrows in callers.  These would have to be addressed on a case by case basis, and there is always the option of reversing the process.

Daklu wrote:

My initial concern is it requires LV to manage a complex network of relationships between packages, namespaces, and code mutations. How are those relationships defined? Who is responsible for maintaining them? My second concern is the wisdom of building package awareness into LV. Currently LV doesn't know anything about packages. Intuitively I think that's a good thing. It helps keep the complexity manageable. My third concern is it might be a viable end point, but we still have the problem of getting there from where we are now.

Maybe I'm missing something with just changing the depenency paths.  The idea is to provide a simple way to relink multiple project VIs from one Child package to another Child in the same package Family.   I'm not convinced that LabVIEW would have to understand VIPs.  In the end, we're just talking about a Family\Child directory hierarchy.  ("Package" here refers to the VIs placed by the VIP, not the VIP itself.)  But, I admit that I'm just brainstorming here.

Daklu wrote:

VIPM could allow package users to dynamically select versions within a Family to install SxS or uninstall.

What do you mean by "dynamically" install or uninstall?

Just meant that VIPM could update the LabVIEW palettes after package installs or uninstalls without restarting LabVIEW (as it does today), only it could do this from within a package-Family-based-UI and with respect to SxS.

Daklu wrote:

If there is a concern that package developers won't create project libraries for their reuse, maybe VIPM should? (At home w/o LV and I can't remember if VIPM does this already.)

No, it doesn't do that currently and I don't think it should. At least not automatically. It would be (in my opinion) one of those features that tries to be helpful but ends up just being annoying and difficult.

If an LVLIB already exists then VIPM should not do anything.  However, if an LVLIB doesn't exist then we've got namespacing issues on the horizon.  VIPM could give a choice of applying LVLIBs or PPLs if a project library doesn't exist or it could just enforce that the package developer create a project library before building the VIP.

Daklu wrote:

At this point I'm thinking the standard sxs model is the only solution that gives us a workable solution.

Thanks for helping us think through the implications of each design choice.

[vishots.com]

Just meant that VIPM could update the LabVIEW palettes after package installs or uninstalls without restarting LabVIEW (as it does today)

VIPM does not restart LabVIEW to refresh the palettes. VIPM already refreshes the palettes in place.

[LabBEAN]

MichaelAivaliotis wrote:

Just meant that VIPM could update the LabVIEW palettes after package installs or uninstalls without restarting LabVIEW (as it does today)

VIPM does not restart LabVIEW to refresh the palettes. VIPM already refreshes the palettes in place.

This is one of my favorite features of VIPM, and I think this "dynamic" behavior could be espcially helpful in an SxS scenario where users can easily adjust which packages are installed while viewing them by package line/Family.

[Daklu]

Maybe I'm missing something with just changing the depenency paths. The idea is to provide a simple way to relink multiple project VIs from one Child package to another Child in the same package Family. I'm not convinced that LabVIEW would have to understand VIPs. In the end, we're just talking about a Family\Child directory hierarchy.

For LV to manage package replacement for us, it has to know something about the package.  For example, after Foo.vi has been installed it has no idea how it got there.  Maybe it was installed via packages, maybe it was installed via windows installer, or maybe somebody copied it there.  How does LV know Foo is even part of a package and which package it belongs to?  Your suggestion has LV figure it out from the directory structure.  In other words, the file system is the interface for transferring package membership and package family information from VIPM to Labview.

That can work, but file system it is not a very flexible interface for communicating information.  How do we extend the interface as requirements expand?  What about all the existing packages that don't use that exact directory hierarchy?  What about packages that are deployed to user.lib?  What about packages that are deployed somewhere else entirely?  What happens if the package developer changes the name of a vi to improve clarity? The goal is to give package users and package developers the flexibility to do what they need to.  Imposing strict directory requirements removes flexibility.  I'd rather have some kind of xml .vipinfo file containing relevant information.

I'm not opposed to eventually rolling this kind of functionality into LV, but I do think it is premature to lobby for it (or plan the mechanics of how it would work.)  Once NI implements something in LV it's hard to change it, so it better be right.  I'm not at all confident we have the best techniques and processes figured out for managing sxs installations.  I'm not even confident we have captured all the use cases.  We're treading new ground here and it will take time to learn how to do things.  Let's focus on the things we can do now, like discovering best practices and creating user tools to do the relinking.  Once they have been established and refined, then we can lobby for LV integration.

However, if an LVLIB doesn't exist then we've got namespacing issues on the horizon.

Namespacing can be achieved by means other than lvlibs.  All project libraries (classes, xcontrols, etc.) allow namespacing, and before project libraries developers appended a namespace to each vi.  I believe VIPM still has that option.  (If the project dependencies branch was able to group by namespace I might use that technique a lot more.)

VIPM could give a choice of applying LVLIBs or PPLs if a project library doesn't exist...

Then you're no longer using standard sxs installations and moving toward custom sxs installations and the problems associated with it.  I do think a custom sxs installer would be a useful tool for end users right now; I'm not convinced it should be part of VIPM or that best practices should encourage its use.

...or it could just enforce that the package developer create a project library before building the VIP.

Nope... don't like.  If putting code in a project library is a best practice, write it up as a best practice... but let the user decide if they want to follow it.  Every time you impose restrictions on a user to "help" them do something right you are removing flexibility another user would find beneficial.

---------------

I'm going to be tied up for a couple weeks and won't get to respond to this thread much, but I'd like to hear what others think about the standard sxs model (every package instance has a different namespace and install point.)  Good idea?  Should we consider it a best practice?  Bad idea?  Does it create too much difficulty for end users?

[vishots.com]

I come back to the original "problem" stated as:

If PackageA and PackageB on the NI Tools Network rely on different versions of PackageC, which may be on the NI Tools network or may be distributed by other methods, A and B will not be able to coexist on the computer at the same time.  This problem cannot be resolved without changing the source code for A or B, which may not be an option and creates problems of its own.

The solution is simple. Package A and B should include package C as internal dependencies. This can be done easily with VIPM Pro, BTW. If you don't want to depend on an external package and want to be isolated from the changes of C, then don't.

I feel that the discussion is trailing off into territory that only the NI LabVIEW core development team can address with changes to LabVIEW itself and is out of VIPM's jurisdiction. In Whatever way NI decides to solve this "problem", I can assure you that VIPM will adapt to support the new ways.