Random Ramblings on LabVIEW Design

Community Browser
Labels
cancel
Showing results for 
Search instead for 
Did you mean: 

Re: I'm not being critical but... (Re-use) Part 1

swatts
Active Participant

Hello my friends.

Hope you are full of LabVIEW loveliness

One of my pet hates is people who say "I'm not being difficult|rude|critical but....." and then proceed to be just that!. Well I'm not being critical but I think the standard model of LabVIEW re-use sucks.

I hate myself!

One of the on-line groups I am a member of is devoted to collecting best practices with regards to  software development using LabVlEW. A subject came up recently with regards to upgrading old code and it became apparent that my company (SSDC) had been working around an issue within LabVlEW practices and we've been doing this for over a decade. The trouble is that when the work-around becomes in-grained it becomes habit and then just something you do. Well my eyes have been opened and I would like to discuss the standard LabVlEW re-use model and how it doesn't fit into our use case.

First I'll describe the standard re-use model as I see it (feel free to chip in and expand this).

Standard.png

In the LabVIEW Environment we have several places to put libraries of re-usable code and these appear in our palettes

\lnstr.lib — Download a hardware driver and it will likely as not appear here

\vi.lib — We park code in the Addons Sub-directory, also toolkits go here

\User.lib — This is the main directory for re-use code

Using the excellent VIPM makes this easier to manage.


So in the ideal world for this model our final piece of software will consist of VIs and Controls pulled from these libraries. Our projects then become an assembly of our own code and this re-useable code. This is part of System Configuration Management and a white paper (especially section 5) discusses this.

Let's call this re-use in the environment.

This is how it has been for as long as I've been using LabVlEW and for as long as I've been using LabVIEW I've been fighting against it.

So what's the issue?


Let me describe some common worries

Upgrading, moving machines, deploying source-code

  • PC goes phut!
  • IT Rollout a new operating system
  • LabVlEW gets upgraded
  • We want to set up a new development system.

Now it's a bit annoying to reload all the dependencies (by dependencies I mean drivers, re-use libraries etc etc), but manageable enough.

Linking to dependencies, lost links, version cock-ups

More seriously for me is the quiet/silent changes.

Lost.png

So we change a system VI and this changes things in other programs, this has happened to me and is incredibly annoying, in de-bugging it's all too easy to open a VI that isn't doing quite what you want it to, but if you change it it will cause issues in other projects that share it.

Don't get me started on multiple versions of LabVIEW on 1 PC and the save-all disaster (I'm sure I'm not the only one!)

Version control, traceability and validation

Here's the thing...I've got my project under version control, version number on the front panel for all to see. TortoiseSVN shows all ticks. I open the General Error Handler.vi and change it, save it. Nothing visibly changes in the project, worse nothing visibly changes in all my other projects. I have no way to detect changes in my project caused by changes in shared library code.


“This might be the last of the Ten Commandments but it’s a really, really important one. When an app has external dependencies which are required for it to successfully build and run, get them into source control”

http://www.troyhunt.com/2011/05/10-commandments-of-good-source-control.html

Storage.png

Now on the assumption that storage space is essentially free now I propose a slightly different model.

Keep re-useable code in the system libs as a local cache, but when they are moved onto the block diagram offer the choice to create an Ivlib in the project. This should protect the name-space, any other tools from the same lib could be added to this Ivlib. This Ivlib should be stored in a sub-directory called Dependencies below the calling VI, give it an auto-generated name like instrlib- Agilent 34401.1vlib and the Project can then contain all of the shared VIs. This will then make a package of standard LabVIEW that can be versioned and stored away. So take NIScope, a fairly large set of VIs even if we're not selective we are only adding 2Mb to our project size. A complex project might add 50Mb.

Proposed.png

Let's call this re-use in the project.

For 100% confidence in your build I suggest you would need 100% traceability of the code elements - there is an argument to include all dependencies in this, I'm talking dlls here! I middle ground suggestion maybe to include a dependency manifest as part of the project files that lists all dependency dlls, versions, date changes etc. This would be updated when the project is opened.

I think this may be an interesting discussion (and not one I've heard discussed much), part 2 will look a bit futher forward and be a bit more radical.

Many thanks to Jack Dunaway, Adrian Brixton and Jon Conway for helping to turn my vague grumblings into a coherent article.

Looking forward to Geneva CLA Summit, not looking forward to re-certifying, maybe this will be my last one!

Lots of Love

Steve


"The only other advice I can relay is that you should write code as if it were mistaken, and you wil..." Tef

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

Comments
FabiolaDelaCueva
Active Participant

Steve,

Excellent post as usual!

I struggled with this at the beginning. Then I discovered how I could manage dependencies with VIPM and I have not looked back.

Some people are not aware that starting on VIPM 2012, one can convert a zip file downloaded from IDNET into a package:

http://support.jki.net/entries/22443418-how-to-import-a-labview-idnet-driver-into-vi-package-builder

Now, once the developper has all the packages needed for a project, they can choose to save all these packages and other dependencies in folder in their source code control and install each package one by one when the project is opened in a new system OR...

They could create a VIPM configuration file. This is one feature available on the VIPM professional version. These files can be applied with the free version. A .vipc file includes all the packages used by a project. There are two options, the package can be included by reference, meaning only the name and version is stored and when the .vipc file is applied then VIPM contacts the repository and downloads the package. However if the developper is not sure they will have access to the internet, then the packages can be included directly in the .vipc file. This makes it really easy to move to another machine and just right click on the .vipc file and choose apply.

more on that here: http://support.jki.net/entries/20700171-Using-VI-Packages-in-your-Projects

Yes creating .vipc files can only be done with VIPM pro, but applying them can be done with the free version.

Looking forward to seeing you again at the CLA Summit in Geneva. You will be missed in the CLA Summit in Austin.

Fab

For an opportunity to learn from experienced developers / entrepeneurs (Steve, Joerg, and Brian amongst them):
Check out DSH Pragmatic Software Development Workshop!

DQMH Lead Architect * DQMH Trusted Advisor * Certified LabVIEW Architect * Certified LabVIEW Embedded Developer * Certified Professional Instructor * LabVIEW Champion * Code Janitor

Have you been nice to future you?
swatts
Active Participant

I too have a strategy to get round it, i.e. I only use the environment libraries if I have to. LabVIEW should be doing this for me is the point I'm trying to make.

Have fun next week and really looking forward to Geneva.

Need to get my ratio of Joke slides to Serious slides about 50:50 and I'll be happy, currently it resembles a slightly surreal stand-up routine.

Steve

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

James_McN
Active Participant

Steve,

You definitely got me thinking about this as I think you are on to something. It is not usual to find customers who are really cautious of having any code change don't consider or are unaware that while you are pointing to these shared folders it could upgrade at any moment, or moving between machines you are now silently using a different version.

I do think a replacement would be difficult to make seamless but not impossible. I'd love to know how some of the text based Environments deal with the same issue, I will have to try and do some research. It is probably more complicated in LabVIEW as reuse code is mostly saved as source.

It seems like it should be possible to do something though, would need to do some research into project providers.

Great ideas as always!

James Mc
========
CLA and cRIO Fanatic
My writings on LabVIEW Development are at devs.wiresmithtech.com
swatts
Active Participant

I think it was the troy hunt article that filled me with some confidence that I was on the right track (IMO), I also think the local cache idea works because it doesn't destroy the existing model, in fact it builds on it.

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

James_McN
Active Participant

Do you have a link? Would be interested to take a look

James Mc
========
CLA and cRIO Fanatic
My writings on LabVIEW Development are at devs.wiresmithtech.com
swatts
Active Participant

It's under the green quote

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

CRoebuck
Member

Great article Steve

VIPC's are great for just this. I used them for hopping between projects (before I started using VM's) and like to think of them as packages of packages........

Local cache is an interesting concept though, one might argue that if you've make a change to General Error Handler (and if it was implemented as a class natively) then you could just call an overide however we don't have such luxuries

See you in Toblerone country

Don't forget to give Kudo's for a good answer !

LabVIEW Champion
Certified LabVIEW Architect
Certified TestStand Architect
swatts
Active Participant

Thank you sir,

The reason I'm not grabbing onto VIPCs is that they are still outside the traceability of the project, so for example I could load a different version of the VIPC and it would not show up as a change in my source-code. That is unless you push the VIPC directly as part of the project. Now the scheme I see is to have the local cache under version control too, but environmental (i.e. not related to any project). But still load any dependencies into the project directory.

I see VIPCs as a system configuration tool rather than a project configuration tool. Unless of course I'm talking nonsense.

See you tomorrow as well as in Geneva

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

Thoric
Trusted Enthusiast

I'm perpetually amazed how you find things that didn't concern me before and now make me lose sleep at night. This is a very important article.

Currently, I use virtualised environments for each inidividual project. With the cost/Gb of storage space as it is (as per your plot), it's not a problem for me to create literally dozens of VMs a year, thus isolating each solution from the other and avoiding some of the headache you identify above. Phew.


However, that's only really effective when I work on my own projects. When in teams, local dependency version conflicts are still a concern.

Thoric (CLA, CLED, CTD and LabVIEW Champion)


swatts
Active Participant

Thanks mate, we will be discussing it at CSLUG tomorrow, I'm hoping to really thrash out the concepts in the next few weeks and that will hopefully lead on to another article. In my head I have a vague idea about what I would really like to see, others have a much clearer idea. The question then is the unintended consequences, what is good for my use case may knacker up another use case. I quite enjoy this kind of stuff.

And still your comment gets more likes than my article, it's sooooo unfair

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

Thoric
Trusted Enthusiast

Shame I can't get to CSLUG, it would be great to join in the discussions. Maybe at Geneva!

Don't judge your articles on the 'likes' count, qualify them on their ability to attract valuable comments

Thoric (CLA, CLED, CTD and LabVIEW Champion)


swatts
Active Participant

The comments are what makes it so much better than book writing, I think I got 2 emails (one of them from Fab) as feedback about the book. I learn so much more from these and if I'm not learning I'll give up writing being a purely selfish creature.

I encourage everyone to spend as much time in the comments as in the main text.

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

wirebirdlabs
Member

Fact and tautology: any distributable dependency that is not in source code control is a dependency that is out of control.

Things that are out of control represent some finite level of risk and liability. How much risk/liability is OK to assume? (That's a context-sensitive business question between you and your stakeholders.)

swatts
Active Participant

it's funny, because it started off as an article about convenience but very soon became more about control and traceability

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

swatts
Active Participant

I presented this to CSLUG (Central South LabVIEW User Group) today and was half expecting some use-case that would destroy the argument (varied group, mainly industrial users). The interesting thing was that no-one seemed to use the User Lib directory and all had various strategys for getting around the issue. Most interesting.....

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

crossrulz
Knight of NI

I assume you'll have another post with a decent summary of your findings?  I'm quite curious since I'm trying to figure out how exactly to use a reuse library in my company.


GCentral
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
"Not that we are sufficient in ourselves to claim anything as coming from us, but our sufficiency is from God" - 2 Corinthians 3:5
swatts
Active Participant

Sadly there is currently no seamless way, but the people I've spoken to internally to NI have been very receptive to the issue. To be honest my aim is to get NI to do something within LabVIEW (obviously this is way in the future, blue sky thinking). Or at least for recognition that there is an issue worth discussing.

The strategies people have consist of the simple (saving as.... protect namespace etc etc), to the more complex using VIPCs and keeping them as part of the project.

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

AristosQueue (NI)
NI Employee (retired)

Congratulations, you have now defined DLL hell. When C programmers speak of DLL hell, they speak of the problem of trying to decide which components use a dependency from a central component that upgrades and gets bug fixes versus which components have their own copy of the dependency that is a known version and won't get new bugs added to it (or even bug fixes that break previous workaounds). When should you have your own copy of a dependency and when should you depend upon the central installed version? That is a question whose answer varies project by project, component by component, as far as I can tell.

It gets worse when two (or more) reuse components get reused together. If both of them have their own copy of the DLL, they may not be using the same version of the DLL. Which one should you use? Can you ever use those components together? Probably not. Same problem occurs if one has its own copy of the DLL and the other is using the central DLL -- they may work together because they happen to be the same version until the day the central DLL gets upgraded.

For myself, I decided long ago that I always leave myself open to being shattered by an upgrade. My personal projects are almost always done in the newest shipping version of LabVIEW. I upgrade as soon as the release is final. Why? Because then my component parts always work together -- I keep everything dependent upon the same version of LabVIEW so that all my personal libraries work together when I reach for them. It is a terrible way to work -- it results in lots of lost time when I upgrade and things don't work anymore. But I no longer see that as a problem to be fixed -- the pain of having to manage separate dependency paths has been, for me, a far greater time sink. In other words, as bad as "keep everything dependent on the central repository" is, it is better than all the other choices I've experienced.

And yet, I recognize that mileage may vary. Other people have had such terrible nightmares about unexpected effects of upgrades that they want their own copy of everything and they'll upgrade only when forced to by the need for a specific new feature or bug fix. That matches their experience. It isn't wrong. It's just where they feel pain.

I don't think there's any non-painful solution. But it is good that this alternative is now in your mental toolbox -- sometimes it is a useful approach.

swatts
Active Participant

Both I and James Mac were interested in the text based language equivalent so many thanks for that. My experience of working in that environment is that you can spend a LOT of time wrestling links and dependencies, fighting with programming colleagues etc etc. It made me enjoy LabVIEW more and more to witness it..

I would say tho' that your use-case is radically different than 99% of LabVIEW consumers. You are very affected by upgrades in LabVIEW because you are making software for LabVIEW. I don't care so much, I very rarely upgrade a version of LabVIEW half-way through a project, there's just no profit in doing that. An interesting observation I had when I presented this at a user group is that very few people actually use User Libs as set out in the linked white paper, most had some strategy for getting around it.

Anyways I've been thinking about how it could be done practically as a software problem. So here's a requirements spec....

  1. On project load update the DLL/Dependency manifest (maybe this could be done when LabVIEW loads and stored away to speed things up)
  2. Monitor when something from an environment library has been dropped onto a block diagram and copy the entire Lib to the project directory dependencys folder, while protecting name-space.
  3. (Now it starts getting complex....) If I drop something onto my block diagram that is already in my project dependency lib but I've used the standard function palette, LabVIEW should know to drop my project version.

I think that when put like this it shows that there might be performance issues to worry about etc etc. I still think it's a valuable conversation but it's quite a tricky job to get right.

I also think it's an important subject. Looking at the time and effort people are putting into SCC, it's then a shame that if you follow advised LabVIEW re-use practices, are very efficient with your re-use code etc 80% of your code could be uncontrolled. (I'm plucking the 80% from old silver-bullet studies about how much re-use you could expect on a perfect system, I'm sceptical but it didn't stop me using it)

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

swatts
Active Participant

The other option I guess is a tool that sweeps a project and takes out dependencies on demand. This I think is possible without digging around in LabVIEW.

Thoughts?

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

samsharp99
Member

Interesting article - it's aimed at 'developer convenience' but has an impact on a large number of topics for software engineers - build configuration management (can I build this exact version of the software (+ dependencies) again in the future), collaborative working (can anyone check-out and build the software if I'm away or ill), traceability (which versions of functions & libraries was I using). I think the fundamental problem with the 'standard re-use' model that you describe is that the contents of the user.lib isn't in source control (not necessarily with the project - but *somewhere*).

Of course, LabVIEW doesn't actually stop you from editing VIs in vi.lib (such as their own function libraries) which is when you *really* start getting into sticky territory as you've got no chance managing your configuration then as all changes will be lost when another dev checks out your code or your PC crashes.

Our team has just started using VIPM Pro - we're starting to convert our re-use libraries into VI Packages and storing a .vipc file with the project in source control. Of course...this only works for things that are (or can be converted into) VI packages. I think that solves a lot of configuration issues - especially when we get to the stage where the generation of the .vipc file is automated at build-time (and the configuration is logged) to check that the .vipc contents match before building.

As an aside - I've had to go through the process of moving to a new development machine in the last couple of weeks - I created a .vipc file on my old machine and then ran it on the new machine and hey presto - all my packages at the same version I was using.

Previously we had all of our re-use code stored in a source control project that we checked out to user.lib. This was great until we moved to a new version of LabVIEW and had to maintain 2 (or more...with LV2013) versions of the library and quickly ran into issues (with updates only happening in one or the other version at different times by different people).

A simple solution perhaps is to have your user.lib, vi.lib and instr.lib LabVIEW directories in source control (as well as your project files - assuming all dependencies are in one of those folders or your labview project)? If nothing else it gives you the power to see what changes have been made?

I think I like what you're proposing though, pulling a copy of *all* of the dependencies into the project (and, as such, source control) - it would certainly make checking out and building others' code much simpler when VI packages aren't available/used and pretty much the dream for doing 'clean (automated) builds' on a virtual machine.

swatts
Active Participant

Welcome Sam!

That summarises nicely how it developed, putting the user libs etc in a system based source control was going to appear in part 2. Without giving away everything I rather like the idea of a cloud based source control system for system stuff, user-lib under the users control, vi lib under NI control, instr lib under user control. This is just mental exercise really at this stage. But if we don't thrash out what we want we can't really complain that it's not been provided.

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

FabiolaDelaCueva
Active Participant

swatts wrote:


                       

The other option I guess is a tool that sweeps a project and takes out dependencies on demand. This I think is possible without digging around in LabVIEW.

Thoughts?


                   

Might not be exactly this, but Becky Linton presented at NI Week 2013 about how to update legacy code to use the new packages created so all the projects would use the same reusable code base, I could imagine something similar could be done for other dependencies. Here is a link to her presentation material:

https://decibel.ni.com/content/docs/DOC-30858

For an opportunity to learn from experienced developers / entrepeneurs (Steve, Joerg, and Brian amongst them):
Check out DSH Pragmatic Software Development Workshop!

DQMH Lead Architect * DQMH Trusted Advisor * Certified LabVIEW Architect * Certified LabVIEW Embedded Developer * Certified Professional Instructor * LabVIEW Champion * Code Janitor

Have you been nice to future you?
FabiolaDelaCueva
Active Participant

Sam,

Excellent use of VIPM and .vipc files. It has made it a lot easier for me to support mutliple developer teams and multiple LabVIEW versions.

That and the addition of having an option to convert installed instrument drivers into packages has made the pain that Steve describes a lot more manageable.

I agree that there are still other dependencies that unfortunatelly are outside VIPM reach like 3rd party non LabVIEW drivers.

Fab

For an opportunity to learn from experienced developers / entrepeneurs (Steve, Joerg, and Brian amongst them):
Check out DSH Pragmatic Software Development Workshop!

DQMH Lead Architect * DQMH Trusted Advisor * Certified LabVIEW Architect * Certified LabVIEW Embedded Developer * Certified Professional Instructor * LabVIEW Champion * Code Janitor

Have you been nice to future you?
AristosQueue (NI)
NI Employee (retired)

swatts: I was using my experience of upgrading LV as an analogy for the general situation. I generally prefer all of my dependencies to be centralized ones when they come from third party sources. I try to never fork my own copies for any dependent piece and instead rev my code on the cadence of my dependencies. In general, dependnecies *try* to not break backward compatibility, so the time I waste when they do is, to me, the reasonable expenditure of time compared to sorting out what happens when many dependencies come together. But, again, it is personal experience that decides this.

Daklu
Active Participant

Steve, your post touches on some issues I highlighted 2 years ago.  Unfortunately it is still a risk that is rarely recognized among Labview developers.  (Though to be fair, it is also a rarely encountered risk... for now.)

swatts
Active Participant

Interesting stuff Daklu,

Another interesting aspect of this discussion is that I was expecting someone to jump in and defend the current model, but it seems to me that no-one really uses \user lib and everyone has strategies to work around it. This is surprising as it appears in all the training etc I've seen that this is the way to do it and also seems to me to be a wasted opportunity.

Thinking deeper about comparisons between LabVIEW and other environments I feel that a difference is that editable environmental code can be a very visible part of the source code (i.e. it just appears on the block diagram). In other languages you explicitly link to re-use libraries by declarations and editing these is a little more difficult. I'm thinking that the next step may be to talk about requirements for a re-use library from scratch. They all seem to run into dll hell and this may be just a visualisation and declaration issue.

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

samsharp99
Member

I think for most LabVIEW users - a lot of these problems don't even register on the radar. I don't know the statistics but I would have thought that the majority of LabVIEW users are engineers/scientists solely working on applications (and maybe not even building them into executables) and not software engineers. I think the issues start to arise when you're looking at developing applications as a software engineer or with a team of developers.

As for defending the current model - it's fine as long as you have nothing in user.lib! (except maybe scripting VIs?) To have control of your source code you need to have control of your dependencies so they need to either be versioned VI packages or in source control! If you're saving VIs in user.lib and using them in your applications (without at least having them in source control) I think you're going to run into troubles eventually (giving your code to someone else, your computer dies or you make a hasty change to a VI and it breaks backwards compatibility).

Daklu
Active Participant

Regarding user.lib, I think NI moved away from that as a location for reuse code sometime around 2010.  If I recall correctly, when I first released LapDog they recommended deploying reusable code in vi.lib.  The only thing I'll put in user.lib is things like templates.

IMO, one of the biggest differences between Labview and other environments is most other environments separate edit-time and compile-time.  In Labview edit-time and compile-time are one and the same.  That decision does provide instant feedback during editing, but it also makes some things harder.  I wish we had a way to modify the edit-time environment without installling/uninstalling packages.

swatts
Active Participant

Sam_Sharp wrote:


                       

I think for most LabVIEW users - a lot of these problems don't even register on the radar. I don't know the statistics but I would have thought that the majority of LabVIEW users are engineers/scientists solely working on applications (and maybe not even building them into executables) and not software engineers.

Speaking personally I ran into these problems on my second project, the LabVIEW 5 solution for us was dumping everything in a llb. The associated risk of doing that was less than the constant battle with cross-linking etc. For me I was just solving an annoyance, but for years I thought I was just going against the flow. I think it has transpired everyone else has been doing it too.

I'm not convinced that pulling things into the environment to affect my project build using VIPM is the right thing to do, although it is better than doing nothing. My view is that the project should be pure as soon as you go through the effort of putting a version number on it. Any changes should show up very visibly.

With regards registering on the radar I think  that LabVIEW masks it by being a very high level language and allowing us to spin up code quick. I think other languages are catching up and this seems to be based on a decent re-use strategy.

This is a really interesting discussion I appreciate all your inputs a great deal, it's the fuel for me.

Many thanks

Steve

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

swatts
Active Participant

I think you summed up what I ran into the moment I tried mentally running through a solution. I still like the model I present (for me!), I can't really think of any negative impact (performance issues aside). I also think a clear re-use model would open the floodgates.

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

AristosQueue (NI)
NI Employee (retired)

Daklu wrote:


                       

Regarding user.lib, I think NI moved away from that as a location for reuse code sometime around 2010.  If I recall correctly, when I first released LapDog they recommended deploying reusable code in vi.lib.  The only thing I'll put in user.lib is things like templates.


                   

No, we have not moved away from user.lib. LapDog does not belong in user.lib because it is a third-party install. User.lib should be for the user's reuse code, not for third-party installs. Third-party installs belong in vi.lib\addons.

wirebirdlabs
Member

user.lib ... vi.lib ... %LocalAppData% ... a networked drive ... these are all generalized to "external linkages not maintained with an ACID workflow alongside app code in a SCC repository." That's the root problem, where comments in this thread have suggested some workflows that are better than nothing, but still not solving the root issue.

Daklu
Active Participant

AristosQueue wrote:

No, we have not moved away from user.lib.

Hmm... maybe NI did a better job of clarifying the message, or maybe the message was always clear but I didn't understand it until I got around to putting LapDog in a package.

AristosQueue wrote:

User.lib should be for the user's reuse code, not for third-party installs. Third-party installs belong in vi.lib\addons.

There seems to be a bit of cognitive dissonance in that statement.  When creating reusable code it is a fairly well-established best practice to keep the source code and deployed code separate, and treat your own reuse code the same as a third-party install.  With that in mind, what would you recommend be put in user.lib?

AristosQueue (NI)
NI Employee (retired)

Daklu wrote:

There seems to be a bit of cognitive dissonance in that statement.  When creating reusable code it is a fairly well-established best practice to keep the source code and deployed code separate, and treat your own reuse code the same as a third-party install. 


                   

Sure, that's all true... if you're an uber-advanced code-reuse jockey with a coherent system of libraries and modules that you use in your projects.

Now, if you're an average LabVIEW user and you have what amounts to a junk drawer of VIs that you use repeatedly on projects, use user.lib.

AristosQueue (NI)
NI Employee (retired)

Jack:

ACID? Acryonym unknown. Googling... found... http://en.wikipedia.org/wiki/ACID

Is this what you were referring to?

Daklu
Active Participant

AristosQueue wrote:

Now, if you're an average LabVIEW user and you have what amounts to a junk drawer of VIs that you use repeatedly on projects, use user.lib.

So it's designed as a convenience for the "Labview as a tool" users rather than for professional developers?  I can roll with that.  (Though it does add to my list of reasons why I wish Labview were two products instead of one.)

AristosQueue (NI)
NI Employee (retired)

Daklu: Almost true... it was designed for both... the definition of the professional has evolved over time. 🙂

wirebirdlabs
Member

AristosQueue wrote:

ACID? Acryonym unknown. Googling... found... http://en.wikipedia.org/wiki/ACID

Is this what you were referring to?


                   

Yes. Treating SCC operations as db transactions and the repo as a db enables reasoning about how the application code and builds will behave. If that database is referring to data outside its realm of control, all bets are off on whether the JOIN operation performed by a build actually contains what you think it should, or behaves as you think it ought.

If we could treat vi.lib (and other sources of external links) as immutable for all time, we're OK; but these external data stores are not immutable, and maybe we shouldn't always treat them as if they were.

We sometimes get away with approximating these as immutable -- but that doesn't scale to larger applications or longer project lifetimes or larger teams.

One negative side effect of approximating immutability is LabVIEW's standard library carries around lots of baggage as an explicit strategy -- "Well, we've identified this defect in this API in vi.lib, but changing it would break existing customers who are linking to it, so we'll keep it." It's a tradeoff we're making with eyes wide open, but it's perhaps not the most desirable tradeoff for all.

ohiofudu
Member

"ACID" Ohoooo lots to learn.

Certified LabVIEW Architect
Certified TestStand Architect
AristosQueue (NI)
NI Employee (retired)

JackDunaway wrote:

One negative side effect of approximating immutability is LabVIEW's standard library carries around lots of baggage as an explicit strategy -- "Well, we've identified this defect in this API in vi.lib, but changing it would break existing customers who are linking to it, so we'll keep it." It's a tradeoff we're making with eyes wide open, but it's perhaps not the most desirable tradeoff for all.


                   

It is a tradeoff you might as well make because NI *must* make that strategy choice. Why? Because we might have some customers who treat it that way. As such, the palette APIs are treated as "we don't break backward compatibility without an overwhelming reason." We fix bugs internal to the VIs, but unless we're able to play the mutation card to keep code working, we try to not break vi.lib. There are several VIs where we have added Recommended terminals that might have been Required if they had been there in the first shipping, and there are other VIs that are deprecated from the palettes because we changed the conpane. Given that we're already doing that calculus, you might as well take advantage of it. At least for LabVIEW Core, anyway.

wirebirdlabs
Member

AristosQueue wrote:

It is a tradeoff you might as well make because NI *must* make that strategy choice. Why? Because we might have some customers who treat it that way.


                   

In steps the beauty of Steve's suggestion in the original document -- copy functions a developer uses from vi.lib (or user.lib, or wherever DAQmx or NI-VISA or packages from the LabVIEW Tools Network choose to install their libraries) into the application directory (which is also, nominally, the local copy of the SCC repo of the app codebase).

This way, the application code links strictly to dependencies only within its own managed repository (rather than an unmanaged repository that we currently know as "vi.lib and friends, In Whom We Trust Remaineth Good and Holy").

This way, any venture forwards or backwards within the app repository (checking out current or previous versions) represents an atomic, contrained, sufficient action to guarantee repeatability in behavior, tests, and builds, given the RTE and build stack of the language are constrained in the environment (which brings our assumptions to parity with expectations in other languages).

Using SemVer rules, vi.lib (and other re-use library) maintainers moving forward could opt to release patches, or major/minor revs, or both, to their libraries. Those libraries (and library dependencies) that were copied into end-user's app repos would maintain symbolic references to vi.lib functions, being aware of patch releases (x.x.N-x) that could safely* copy over and replace the old version in the app repo (*ostensibly), and aware of new releases (x.N.x-x and N.x.x-x) that might break existing interfaces requiring some dev time to re-integrate, yet providing substanstially more desirable features or fixes.

Said another way -- vi.lib becomes the intermediate cache that it ought to be. A convenience, connecting developers of re-use libraries and end-users of those libraries, and a decent starting place for dev environments.

The dev lifecycle of LabVIEW's standard library (which we lovingly refer to as vi.lib) could be asynchronized from that of the LV IDE and RTE... for better or worse... tradeoffs, etc... cue tomatoe throwing etc... Though. Still consider. Professor Watts' suggestion of copying libraries from the standard library into the app directory I believe provides room for the legacy practice to remain, while allowing a new, better practice to emerge.

I reckon it also could provide incentive for standard library developers to consider the best practice of using VIPM for re-use code distribution 🙂

PS - I'm curious how regulated industries today even account for today's current LabVIEW practices of linking to unmanaged source code within vi.lib. It'd be one thing if vi.lib were hashed and integrity validated...

PPS -- I think pedantic/academic conversations along these lines are not particularly valuable, and oftentimes even hurtful. This topic far from pedantic/academic -- this issue has caused Significant Heartache in my own LabVIEW Software Engineering endeavors, and I don't see obvious/simple solutions in the near future, though i've got some ideas worth bouncing around with those interested.

swatts
Active Participant

I'm unlikely to write a part 2 to this as you clever people have written it better than I ever could.

That SemVer link is worth a read.

JackDunaway wrote:

provides room for the legacy practice to remain, while allowing a new, better practice to emerge.

I think this may be the key observation for me, I think everyone has been working round this and it has been inhibiting best practice. I personally would start building and collecting re-use code if it was risk free for my existing projects. I have a very low tolerance of managing software when I could be designing software.

The other aspect of this is that if these libraries are not immutable then it devalues Source Code Control, Verification and Validation of the produced software. There's markets for software out there that we have trouble playing in and we will continue to do so unless we get a grip on this.

I'd like to thank all contributors again, your efforts make this worth doing.

Steve


Opportunity to learn from experienced developers / entrepeneurs (Fab,Joerg and Brian amongst them):
DSH Pragmatic Software Development Workshop


Random Ramblings Index
My Profile

wirebirdlabs
Member

swatts wrote:

I have a very low tolerance of managing software when I could be designing software.

Nailed it.

AristosQueue (NI)
NI Employee (retired)

JackDunaway wrote:


                       

swatts wrote:

I have a very low tolerance of managing software when I could be designing software.

Nailed it.


                   

This is a part of why I hope never again to sign my own paycheck. I want to be part of a staff where someone else is handling the library management. LV R&D has people who *enjoy* that aspect -- the version control, the compatibility testing, the license analysis, all of it. I ask if I can use something, they say yes or no, and if they say yes then, thereafter, from my perspective, magic happens and it just stays working. And if it stops working, it's because something substantial has occurred that actually needs design attention.

CharlesB64
Member

I've always kept re-use components out of user.lib, precisely for version control concerns. Quite frustrating, since I don't see them in my palette, and always have to go to project window when I want to pick a VI...

Has anyone used the LVH Build tools? It seems to ease  syncing of a re-use project under user.lib, which might help in this use case.

The ideal thing for me would be, for a project that uses a versionned component (either in-house or external), to have it specified in some file under project's directory, and on loading the project LabVIEW would know where to pick the various components, and put them into user.lib. Something like python's virtualenv, for those who know it.

FabiolaDelaCueva
Active Participant

CharlesB64 wrote:


                       

I've always kept re-use components out of user.lib, precisely for version control concerns. Quite frustrating, since I don't see them in my palette, and always have to go to project window when I want to pick a VI...

Has anyone used the LVH Build tools? It seems to ease  syncing of a re-use project under user.lib, which might help in this use case.

The ideal thing for me would be, for a project that uses a versionned component (either in-house or external), to have it specified in some file under project's directory, and on loading the project LabVIEW would know where to pick the various components, and put them into user.lib. Something like python's virtualenv, for those who know it.


                   

right clicking on a vipc file and selecting "Apply Configuration" opens VIPM and installs all the packages and appropriate versions wherever they are supposed to be installed at: vi.lib, user.lib, etc.

For an opportunity to learn from experienced developers / entrepeneurs (Steve, Joerg, and Brian amongst them):
Check out DSH Pragmatic Software Development Workshop!

DQMH Lead Architect * DQMH Trusted Advisor * Certified LabVIEW Architect * Certified LabVIEW Embedded Developer * Certified Professional Instructor * LabVIEW Champion * Code Janitor

Have you been nice to future you?
CharlesB64
Member

Thank you Fabiola, indeed you can reference packages in .vipc, (in my memory it was only possible to include their whole content).

So the "only" thing we miss now is an integration of vipc at the project level, and a LabVIEW warning at project load you when project's vipc isn't in sync with currently installed packages.

samsharp99
Member

Which I think is something you can easily do with LV scripting and the VIPM API...

2014-03-17_12-53-51.png

Of course, I think this is a VIPM Pro feature but you could script a check to see if the VIPC file matches the installed packages when you open the project and could also auto-generate the VIPC file on save/commit etc.

GregPayne
Member

Good topic Steve, I think you might have just opened a can of worms....

This is something that I have only just come across. The project that I am currently working on has been worked on by many people at different times. There is no library file so all the vi's are in a single folder and one .llb. One of the reasons its all still on LV2010 is because they don't want to 'break' dependancies. Good but sometimes frustrating challenge working through it all.

Chris/Richard: Did I understand correctly that you use a separate VM for each project? Do you know if using multiple VM's would have any single user license implications?