Which Unit Testing Tool are you using?

[A.E.M]

Thanks for all your inputs!

I got some time to get one step further in my setup. I ended up with installing VI Tester and InstaCoverage Pro, as NI UTF seems not really to be recommended and I'm not sure if NI is still actively developing on it. In addition, I need to framework to support OOP.

For now, based on a simple test with a small project, I ended up using InstaCoverage. It is pretty easy to use and to setup new test cases, and auto-generate most of the code which I really like. I felt VI Tester where not as easy to create new test cases and more manual work was needed. I need a Unit Testing Framework which takes as little effort as possible to use and create test cases in, otherwise it becomes a tedious  task that no one really want to spend time one.

I did however find it a bit funny that InstaCoverage really highlight the "Code Coverage" feature as a big thing. I like the idea, but it can really only tell if you have 100% Code Coverage or Not. That is not really so useful for me, as 100% code coverage is not my goal here, but I would have loved to have a metric that I can improve on our time

So with my Jenkins setup I can now clone my Github repo, do a LabVIEW mass compile to make sure all VIs are executable and run the associated unit tests 🙂 Everything happens automatically as soon as I push a new commit to Github.

My next step is to configure a Jenkins file, maybe setup different cases for daily / weekly / release builds 🙂

But for now, I will work more with InstaCoverage and see how it performs in the long run.

Again, thanks for your inputs.

[bokorpet]

---

@A.E.M wrote:

I did however find it a bit funny that InstaCoverage really highlight the "Code Coverage" feature as a big thing. I like the idea, but it can really only tell if you have 100% Code Coverage or Not. That is not really so useful for me, as 100% code coverage is not my goal here, but I would have loved to have a metric that I can improve on our time

It's true that no exact percentage is shown if the coverage is not 100%. Note that, in this case, the InstaCoverage GUI will display the list of uncovered diagrams and will navigate the user to these diagrams in the code. So, in a way, the percentage is given implicitly. The exact coverage percentage display will be added soon. Also, coverage measurement will be optional and configurable. Until then, you can use the InstaCoverage API to produce custom aggregated test results (e.g., ones that ignore coverage).

[A.E.M]

---

@bokorpet wrote:

---

@A.E.M wrote:

I did however find it a bit funny that InstaCoverage really highlight the "Code Coverage" feature as a big thing. I like the idea, but it can really only tell if you have 100% Code Coverage or Not. That is not really so useful for me, as 100% code coverage is not my goal here, but I would have loved to have a metric that I can improve on our time

It's true that no exact percentage is shown if the coverage is not 100%. Note that, in this case, the InstaCoverage GUI will display the list of uncovered diagrams and will navigate the user to these diagrams in the code. So, in a way, the percentage is given implicitly. The exact coverage percentage display will be added soon. Also, coverage measurement will be optional and configurable. Until then, you can use the InstaCoverage API to produce custom aggregated test results (e.g., ones that ignore coverage).

---

Thanks for the info! And great work on the InstaCoverage tool 🙂

Can you tell anything about the roadmap: Are you still planning on maintaining the tool?

[bokorpet]

Thanks for the info! And great work on the InstaCoverage tool :-)

Can you tell anything about the roadmap: Are you still planning on maintaining the tool?

---

Thanks 🙂 InstaCoverage is being actively maintained, supported and improved. We plan various new releases for 2019 and 2020. The next major feature updates will include adding auto-generated wiring in the harness VI, more flexible configurations (coverage etc.) and support for project coverage (beyond VI coverage that is already supported). Parallel to improving on InstaCoverage for the current generation of LabVIEW we're also implementing the updates for the LabVIEW NXG version of InstaCoverage. 

[A.E.M]

---

@bokorpet wrote:

Thanks for the info! And great work on the InstaCoverage tool :-)

Can you tell anything about the roadmap: Are you still planning on maintaining the tool?

---

Thanks 🙂 InstaCoverage is being actively maintained, supported and improved. We plan various new releases for 2019 and 2020. The next major feature updates will include adding auto-generated wiring in the harness VI, more flexible configurations (coverage etc.) and support for project coverage (beyond VI coverage that is already supported). Parallel to improving on InstaCoverage for the current generation of LabVIEW we're also implementing the updates for the LabVIEW NXG version of InstaCoverage. 

---

Awesome! Sounds like it is worth spending more time on InstraCoverage and start implementing it into our organisation 🙂 Do we buy licenses directly from you or NI?

[Dhakkan]

---

@A.E.M wrote:

I did however find it a bit funny that InstaCoverage really highlight the "Code Coverage" feature as a big thing. I like the idea, but it can really only tell if you have 100% Code Coverage or Not. That is not really so useful for me, as 100% code coverage is not my goal here, but I would have loved to have a metric that I can improve on our time

Just a quick caveat... 100% code coverage does not mean 100% test coverage. The former simply implies that each path (wire) of the block diagram of a VI Under Test has been exercised at least once. The latter, on the other hand, implies that the VI Under Test has been examined for all combinations of inputs. Therefore, 100% test coverage implies 100% code coverage.

Depending on the input data types and their count, 100% test coverage can easily become impractical. Therefore, input data vector is determined by balancing between unit testing speed and having a defect-free VI Under Test, subject to 100% code coverage at a minimum.

Vishak

[A.E.M]

---

@Dhakkan wrote:

---

@A.E.M wrote:

I did however find it a bit funny that InstaCoverage really highlight the "Code Coverage" feature as a big thing. I like the idea, but it can really only tell if you have 100% Code Coverage or Not. That is not really so useful for me, as 100% code coverage is not my goal here, but I would have loved to have a metric that I can improve on our time

Just a quick caveat... 100% code coverage does not mean 100% test coverage. The former simply implies that each path (wire) of the block diagram of a VI Under Test has been exercised at least once. The latter, on the other hand, implies that the VI Under Test has been examined for all combinations of inputs. Therefore, 100% test coverage implies 100% code coverage.

Depending on the input data types and their count, 100% test coverage can easily become impractical. Therefore, input data vector is determined by balancing between unit testing speed and having a defect-free VI Under Test, subject to 100% code coverage at a minimum.

Vishak

---

Good point. You are absolutely correct.

[bokorpet]

Awesome! Sounds like it is worth spending more time on InstraCoverage and start implementing it into our organisation 🙂 Do we buy licenses directly from you or NI?

---

It's your choice.

For the coverage discussion: yes, code coverage (or "branch coverage") does not imply bug freedom. "Test coverage", as Vishak put it, is a much stronger guarantee. Tools for establishing such strong guarantees are usually called exhaustive search or model checkers and are mostly academic and/or do not scale. Another coverage metric (stronger than code coverage, weaker than exhaustive input coverage) is "path coverage", which not only makes sure that every branch is taken by at least one test, it also considers the combinations of the branches.

"Project coverage", the one InstaCoverage plans to add next, is another pragmatic metric, which measures that every diagram in the project is tested by at least one test execution (note: the test execution can be a sub-VI call from another VI under test etc.).

[kkoenigco]

Alex,

Upon reading your post, I looked into the LabVIEW CI Tutorial (Jenkins/GitHub) (http://www.ni.com/tutorial/55231/en/).  The tutorial, however, does not specify how to configure Jenkins to use NI UTF.  I am interested in implementing InstaCoverage Pro as you have indicated you did in your post.  Would you mind outlining the steps to configure Jenkins to use InstaCoverage?

Thank you,

Kamalu

[bokorpet]

Dear Kamalu,

here is a draft page of how Jenkins can be set up with InstaCoverage Pro: https://github.com/IncQueryLabs/LabVIEW-knowledge-base/wiki/Continuous-integration-with-InstaCoverag...

We're working on the page at the moment to add more details and polish the example but it may help you already as it is.

Best,

Peter