Command Line Tools for Continuous Integration

[James_McN]

I mean the actual code repository seems to still just be groovy scripts for the web based builder?

My feeling would be that we should have a separate toolset. The CLI tool serves a very specific purpose and I would like to see a separate toolset for build tools that leverage it.

I think from what I'm hearing the best approach maybe to start with what you have, a set of tools for common steps which would also provide a kind of consistent structure to make it easy in the future.

Based on what you have described (not what I have tried so far so correct me if I'm wrong!) there are a couple of things on top of the VIs that I think would make it much easier:

1. A consistent/reliable installation location - I know right now you have to map them to a drive, we have not discussed why, but this would be a nice step to remove.
2. A simple command line API. You have abstracted that to the groovy scripts but in order to run it from any environment I'm thinking you install a batch file alongside each VI which can manage calling it through LabVIEW CLI so perhaps you can get to lv-build-vitester my.lvproj as the command (although that is over simplified!)

This can then act as a template for anyone that needs custom steps too. For example I may want to update the build number in my version constant, it wouldn't go in the tool but I can use it as a template.

Thats what I picture as my dream solution in my head anyway, happy to receive feedback on the concept (and I'm going to work with what you have more this week to solidify it)

[MattP]

Ah, got it.  FYI, there are two branches in there: lv-cli (using your tool), and lv-ci-service (the previous approach using web services that we ditched).  lv-cli is the active branch.  I'm keeping the other one around just as a backup.  Since master was still showing the lv-ci-service info, I just merged lv-cli into master to eliminate that confusion.

FYI, I'm working with the LabVIEW R & D guys on possible future implementations of a CLI for LabVIEW.  They're looking at our discussions pretty closely.  The feedback is being heard!

Also FYI, we also had a custom script file that was consumed by a monolithic LV program to do multiple build steps.  We moved away from that approach in favor of putting the orchestration logic into Jenkins.  This permitted parallel builds much more effectively, as well as running tests on multiple versions.

Thoughts:

1.

The L:\ drive approach we have is actually not necessary - L:\ could be replaced with C:\Users\Public\Documents\LV-CLI Common Steps\Steps\ and work fine.  I used an L: mapping because that's just the process we happen to use on our build server (Jenkins agent lives at J:\, LabVIEW pieces live at L:\, github repos mapped to G:\, just to abstract out where individual systems actually store things), and the package was just a straight export of what we use on our build server. I think the cleaner approach would be to use a system symbolic link to the public documents directory and use a relative path from there.  The reason we used public documents rather than install to a particular LV's vi.lib is that we wanted it to work without modification on different LV versions on the same machine.  Quite probably there is a more efficient solution for that.

2.  Honestly, I'm not sure how much value a batch script would have, since all it would do is call the command line.  You'd still need a wrapper to get the values from another interface and format them to match what the command line needs.

[chinghwayu]

Here's my set of command line tools: https://github.com/chinghwayu/Command-Line-Tools

I added a couple more things in there such as suppressing dialogs and injecting a build number from the CI system.

I submitted mine to the Tools Network about the same as Matt right before NIWeek and I'm happy to deprecate this.

I don't see value in including batch files either but we could have separate template/example areas in the Github repo - common (batch, bash, etc.) and then CI system specific (e.g. Jenkins groovy templates, Gitlab yml, etc.). You don't necessarily need to use/install them but it's there as example implementations and a central place for additional community-contributed documentation.

[chinghwayu]

One utility which I discussed during my session which should be included is the ability to execute specific tests within UTF or VI Tester. Currently, the implementation is basic and only points to a single project file but you may want to run unit tests for specific objects as an example (shorter smoke test) before you execute the entire test suite.

[James_McN]

The main thought process behind the batch files would be to make it as easy as possible for someone to get started as well as making it easy to run steps locally as well but regardless that is a small part - getting the steps right in the first place is the critical piece.

Supress dialogs is great! I'm looking at your build number injection as well. For my own curiosity (as I know you have done the jenkins training) do they recommend this? I have done this on projects but not sure how solid it is since if you lose and recreate the jenkins job the numbers reset.

[MattP]

On the DCAF front (having lost and rebuild build servers a few times now), we use the following strategy for build number injection:

We take the Jenkins build number, the number in the VIPB file, and the build number of the latest deployed package in our internal repository, take the highest, and increment.

The reason we don't use just the Jenkins build number is multibranch builds.  If I have two branches of the same repo, each branch gets different build numbers, starting at 1.  We got conflicting package versions when we just used the Jenkins build number.

[chinghwayu]

Perhaps a VI which allows a user to select an operation with input parameters and displays the resulting command would be a good tutorial interface. A user could then press a button to launch the process. If they like what they see, have another button that allows you to copy the command into the clipboard so it can be easily used elsewhere such as pasting into a Jenkins job.

It's a common build practice to use a number from either the build system or version control where the unique identifier from commits is used.

With build systems, you should be able to specify the next build number to use. With Jenkins, this is available through the UI or API.

There's two ways to deal with multibranch:

1. In Jenkins, you can have a groovy script find your branches, take the highest build number among them, and increment.

2. Branches could have unique versions by incrementing the PATCH number. When it's rolled up to master or the integration branch, increment the MINOR number.

[dramonpr]

Hello Joerg,

I am currently working on university project trying to show the importance of CI/CD and how it can be implemented using GitLab. A part of the project is collecting simple examples of CI/CD pipelines for different languages and scenarios. So far I have created them for C++, Python and JS. However, I have not been very successful trying to implement it for LabVIEW. I am not very familiar with the language, and since everything is windows based I do not know how to get my runner to even compile a VI.

Could you maybe help me? Some inspiration from your code may just be enough so I can implement my own test case.

Thank you very much,

David

[joerg.hampel]

Hey dramonpr, send me a PM and let's see if I can help you.

[dramonpr]

Hey Joerg,

Thanks for your reply. Unfortunately I cannot send private messages since I am newcomer member, and I need to wait 14 days until I am allowed to do so. Maybe if you send me one I can start communicating with you? 

Thanks again,

David