Moving from svn to git

[cirrusio]

So, the title here is deceptive and the question more focused.  A long time ago, I made an unfortunate decision to go with a monorepo model for repository management using subversion.  I have been in the process of migrating my code from svn to git and want to unwind it so I am no longer using this monorepo model as git is not ideal for monorepositories (and I don't want users to have to check out the entire repository if all they want is one component). 

Unfortunately, there is a lot of reusable code that is used across projects (good) but is referenced based on file location in individual projects (bad).  I have identified a couple of ways to address these issues.  The first is to use git submodules.  This would allow a developer to replicate the structure of the repository as necessary, but would lead to duplication if working on multiple projects referencing the same submodule.

The second method is to have a script that wraps git calls so that when any git action is performed, it recursively goes through and makes sure that 1) the directory structure is intact and 2) all files are at the version they are supposed to be.

Has anyone else dealt with this before?  Does anyone have any suggestions?

Thanks.

[cirrusio]

---

The second method is to have a script that wraps git calls so that when any git action is performed, it recursively goes through and makes sure that 1) the directory structure is intact and 2) all files are at the version they are supposed to be.

---

Google's repo tool looks like it performs this kind of functionality in a much more elegant way than my set of python scripts...

[dmysliwiec13]

Hello cirrusio, 

Thanks for the detailed explanation! May I ask what NI products you are using in this application?

Danielle

National Instruments

[cirrusio]

Thanks, Danielle.  So, the reason this comes up is that this is not a single application but a directory of applications, hence the term monorepo and a need to disentangle everything.  These applications span the gamut from just vanilla LabVIEW to LabVIEW pulling in external libraries such as OpenG to applications using RT and FPGA.

[cbutcher]

Could you consider the possibility of using separate repos with VIPM packages for each component?

Convincing your projects to search in user.lib/VI.lib/whichever directory rather than your current structure will likely be frustrating bt with some care definitely doable and then version management becomes much simpler.

Git (or at least, github) allows releases based on git tags which can have binary files such as the vip packages (pretty sure this is like PIN number or ATM machine, but you get the point) attached.

I've done this for a few projects within a collection and found it very useful, especially when it comes to updating another computer (for example, because code is developed on a workstation but executed on a laptop in a lab...)

[cirrusio]

 VIPM is a disaster.  I hit a wall with this a while back and gave up - kept throwing errors and I am not interested in paying for a package manager as I am not trying to do major redistributions.  Would love to see NI someday actually throw their weight behind something like a full up package manger with CLI support.  VIPM is just too cumbersome; we need something like npm or pip.

I think the repo tool is more what I am looking for - basically I am looking for a way to do git submodules without the half-baked submodules routine.

[cbutcher]

If you don't want to use VIPM, you could consider building paths using something like the 'Default Data Directory' or similar as an 'installation' (via copy-paste or more likely in this situation git clone) path.

If you want your (sub) repositories to be independent, you presumably shouldn't require a directory structure that is based around a 'parent' structure. Although my original solution was to install your libraries/modules in user.lib, the default data dir should work in the same way and it's easier to just clone your git repos into "C:/Users/<user>/Documents/LabVIEW Data/<repo name here>/" than Program Files (or similar on another platform, I guess). 

[Jacobson-ni]

Maybe you could try asking some people in the DCAF community?

http://forums.ni.com/t5/Distributed-Control-Automation/gp-p/5235

The framework and modules are all on GitHub so the people working on that project may have some suggestions, although I think it is handled similarly to what cbutcher had suggested.

[cirrusio]

Sorry, I am not following your 'parent' structure comment.  If you can teach me how to code such that LV is completely agnostic with regards to the directory structure, that would be great.  But, to be honest, I am not entirely sure how that is possible.  For instance, when I place code in two places, directory A and directory B (both at the same level), and the code in A references the code in B, LV will always look at that same level for the code in B when opening A.  If I the code in B to some place else, viola!, my code in A pops up with all kinds of missing VI errors.  And that makes sense to me.  So the absolute path is not of central importance (I manage other projects that use gh and don't have to duplicate absolute paths to get everything to compile), but the relative paths are. 

And that is what I need to duplicate so that everything does not break.  I don't really care where a user puts the code they have pulled, I just need to make sure that A and B maintain their relative placements.

[cirrusio]

Thanks for the reply, Matt.  DCAF is a unified framework much like AF is.  The use cases here are completely different.  Whether you pull all of DCAF or just pieces, all work toward the framework implementation and are not likely to be used in another framework.  I will give a quick example of how this works.  A simple illustration of the directory structure goes something like this:

- Instrument

|-Instr0

|-Instr1

|-Instr2

- Devices

|-Dev0

|-Dev1

- Reusable

In this illustration, I have three main directories - Instrument, Devices and Reusable.  Instruments are kind of top level code that describe whole instrument functionality.  Devices are individual pieces that make focused measurements and can be utilized by larger structures to make a holistic measurement.  Each Instrument code base may utilize some of the Devices as well as code in the Reusable folder (just a generic folder to catch support code that on its own doesn't do much).  Devices may make use of the code in the Reusable folder but should not touch be dependent on the Instrument code.  The code in Reusable should not be dependent on either of these other directories.

This is the fantasy.  In reality, all of the code grew up in a very organic manner and I am not sure if separation of concerns is this straight-forward.   But, we shall forward on as if it is (yay!).  Now, if I have a user who wants one of the Devices for their own work, they may pull the  specific Device folder as well as any code required from the Reusable folder.  We could do this using submodules, but this is problematic.  Consider the situation where later the user want another set of Device code.  If we are using submodules, then we end up duplicating code.  And to make matters worse, we can really hose things up if we don't understand how submodules work (they actually point to a specific hash) and  all kinds of funny things happen if you don't update properly.

So, hopefully this makes things a little clearer.  Let me know if you have any other questions.