Unit Testing Group

cancel
Showing results for 
Search instead for 
Did you mean: 

Unit Test Tools in LabVIEW

Great thinking, I had intended to ask a question regarding test on other targets and it completely slipped my mind.

Keep those ideas coming, I am happy with stream of consciousness feedback.

Unit testing code with hardware depenencies will end up being a common use case with LabVIEW.  Test doubles for hardware abstraction imply something about the way the software is written.  This is the thread of thinking that got me writing the unit test blog post, does the ability to write good unit tests require code that was written with testability in mind?

At some point the creator of a unit test tool will have to place the onus of writing testable code on the developer.  One thing I would like to understand is where that boundary exists.

0 Kudos
Message 11 of 31
(3,297 Views)

Speed Speed Speed.

I made my own because NI's unit test is way to slow.  Mine is very very basic but so far works.

https://decibel.ni.com/content/groups/ast-unit-tester

Dan Shangraw, P.E.


   

0 Kudos
Message 12 of 31
(3,297 Views)

Christian,

I agree with your ideas. Excuse me while I hijack your post and expand on some of your points:

kjeld wrote:

  • When creating a UT, the UT framework should auto generate a set of test vectors for the VI under UT

I agree and to take this a step further, the UTF should make it more "discoverable" how to create test vectors. It turns out they are not created from within the unit test definition window, nor by right clicking on the VI itself (where the other unit tests menus are)... one has to go to right click on My Computer or a library or a virtual folder in the project explorer, click on New and then Test Vectors. I know plenty of advanced LabVIEW developers who didn't know one could generate input/output vectors for hundreds of points.

kjeld wrote:

  • Test speed is crucial for executing all UT’s

Agreed, in the "Art of Unit Testing" by Roy Osherove in his second edition book.

Unit Test definition according to Roy Oshervore (emphasis is mine):

"An automated piece of code that invokes the unit of work being tested, and then checks some assumptions about a single end result of that unit. A unit test is almost always written using a unit testing framework. It can be written easily and runs quickly. It's trust-worthy, readable, and maintainable. It is consistent in its results as long as production code has not changed."

If Unit Tests don't run fast, nobody will keep running them and if nobody keeps running them they are no longer trust-worthy, they fall in disrepair.

I wanted to point out that some of your ideas are already there for both Unit Test Framework and JKI VI Tester

kjeld wrote:

  • One button for executing all Unit Test (UT) in LabVIEW project explorer

When UTF and JKI VI Tester are installed, these buttons are added, one needs to enlarge the Project Explorer window to show them (I know they went unnoticed by me for quite a while ) :

UTF and JKI VI tester Run Buttons.png

kjeld wrote:

  • Integrate UT as a native part of LabVIEW project explorer
  • VI’s that are Unit Tested should have an overlay icon for passed or failed
  • Test coverage report for VI’s under UT, also for Reentrant VI!

There are a couple of right click menu options within the project explorer for UTF. And the overlay icon is only related to the unit test (UTF).  In the case where one might have more than one unit test associated with the same VI, I like to have the overlay over the actual test and not the VI itself. The VI below is marked as reentrant VI and I noticed on the coverage report that it was not listed. As soon as I put an extra VI with its tests and that VI was marked non reentrant, then I got the code coverage, this sounds to me like a bug and I had never noticed. I will investigate that further and report to NI my findings.

UTF project explorer integration.png

kjeld wrote:

  • API for continuous integration test, executing and controlling all UT

There is an API for CI for UTF:

UTF API.png

Hope this helps,

Fab

For an opportunity to learn from experienced developers / entrepeneurs (Steve, Joerg, and Brian amongst them):
Check out DSH Pragmatic Software Development Workshop!

DQMH Lead Architect * DQMH Trusted Advisor * Certified LabVIEW Architect * Certified LabVIEW Embedded Developer * Certified Professional Instructor * LabVIEW Champion * Code Janitor

Have you been nice to future you?
Message 13 of 31
(3,297 Views)

jon_mcbee wrote:

What do you consider a "unit" in your unit testing?

There is no one-size-fits-all answer to this, which is why it's called "unit" testing, and not VI, function, subroutine, ..., testing. It depends on teh motivation to test - is it's a lone engineer that wants to lower the anxiety about their or others' code? Is it a regulated space where the FMEA drives testing? Is it CI where code that hase been untouched for multiple release cycles starts acting up?

jon_mcbee wrote:

Do you work in a regulated industry, and if so how does that impact your unit testing?

Yes, and yes. The NI UTF is the only unit test platform that is provided by a company that is ISO9001 certified (I'm sure others are compliant, but not certified AFAIK?)  This means that if I want to include unit testing (and the results) in a regulated project, I either have to verify a non-certifed platform, or use the UTF. UTF has its quirks (and it's fair share of bugs too), but that certification means I'm going to use it. And, because we have the tribal knowledge of UTF, we typcially use it for non-regulated projects too, rather than learning every new platofrm that comes along.





Copyright © 2004-2023 Christopher G. Relf. Some Rights Reserved. This posting is licensed under a Creative Commons Attribution 2.5 License.
Message 14 of 31
(3,297 Views)

ASTDan wrote:

Speed Speed Speed... https://decibel.ni.com/content/groups/ast-unit-tester

Yep Yep Yep. UTF *is* slow, and if your platofmr is slow, people simply aren't going to use it. Or they'll use it far less than they should. Or they'll only use it formally (once at the end of a release cycle), rather than as a CI task (and I'm not necessarily talking about a formal CI server that's running it, I'm talking about running it ad-hoc: more often than your process requires, just because.)





Copyright © 2004-2023 Christopher G. Relf. Some Rights Reserved. This posting is licensed under a Creative Commons Attribution 2.5 License.
Message 15 of 31
(3,297 Views)

FabiolaDelaCueva wrote:

Yep - It's probably ni surprise that I agree with everything Fab (the Queen of Unit Testing) said.





Copyright © 2004-2023 Christopher G. Relf. Some Rights Reserved. This posting is licensed under a Creative Commons Attribution 2.5 License.
Message 16 of 31
(3,297 Views)

jon_mcbee wrote:

What do you consider a "unit" in your unit testing? 

I used to go with the definition in LabVIEW that a unit was a VI. I like now to follow more Roy Oshevore's definition and focus on a Unit of Work. The unit of work could be a VI, a group of VIs, the public API of a class, the public VIs of a project library.  This has helped me design my unit tests better and encourages me to design the unit test first and then create the code to make the unit test pass. It turns out that one discovers a lot more bugs this way that could have come back to hunt later.

jon_mcbee wrote:

Do you work in a regulated industry, and if so how does that impact your unit testing? 

some of my customers do. They have to report code coverage and show that they can tell with confidence whether a change on a vi will impact other parts of their code or not.

The other impact is in the unit testing tools we can use. We are pretty much forced to use NI Unit Test Framework, because it is developed by an ISO certified company and therefor we don't have to go and validate that the tool does what it says it does.

When we are not working with a customer in regulated industry and/or code coverage is not important to them, then we use JKI VI Tester. We can create unit tests for VIs in classes, the tests can be executed at the press of a button and they can help when debugging code. The drawback of the JKI VI Tester is that now one could be inserting a bug in the code written to validate our code. And creating the tests involves a lot more steps that it does for the UTF. You can see the contrast on Unit Test creations on the videos attached to the presentation I gave at NI Week: https://decibel.ni.com/content/docs/DOC-39109 (scroll down and the shorter videos with the demos are there).

jon_mcbee wrote:

Hi kjeld,

Do you use test doubles in your unit testing practice, and if so how do you implement them?

This was a new term for me, but once I went to the link I can say we do use them. Once one starts using real world hardware, connections, databases, etc. It is no longer unit testing and it is integration testing. That said, the same tools used for unit testing can be used for integration testing. The advantage of focusing first on the unit testing is that the bugs get figured out long before one integrates with the outside world. If the unit tests are trust worthy and a bug is discovered, the likelihood of the issue being on the communications side of things is greater and one can focus on that. Instead of trying to figure out what is wrong with the unit itself.

In the case of Hardware drivers, we abstract the actual communication part. For example, if writing a serial communication driver, there will be a class for the hw implementation and one for the simulated implementation. The VISA write and read would be in the write and read methods for the hw implementation, but not in for the simulated implementation. I trust NI to have tested their VISA write and read functions correctly, so I focus on testing that the protocol parameters are formed correctly and that responses are parsed correctly. Once that is figured out, integration testing should just work

jon_mcbee wrote:

How do you use the test results, do you save them or do you mine the data from the files for use in a tracking tool?

Unit Tests are executed before implementing any new features or modifying existing code. Everything should pass. Then a unit test is designed to validate the new feature/change. This test should fail. The rest of the developing time is spent writing code to make the new test pass without making the first set of tests fail. We know we are done when all the unit tests pass.

If multiple unit tests are designed, then executing the unit tests can be used as a measure of progress: We are 80% if 80% of the unit tests pass.

Unit tests become a tool we use to write better code and are executed frequently and kept up to date in our projects. They are exectued before building and deploying. They are our insurance, the reports are just used to get a snapshot of where the code is at any given moment.

The reports are used more in regulated industries to prove that the testing was done and track the results.

Regards,

Fab

For an opportunity to learn from experienced developers / entrepeneurs (Steve, Joerg, and Brian amongst them):
Check out DSH Pragmatic Software Development Workshop!

DQMH Lead Architect * DQMH Trusted Advisor * Certified LabVIEW Architect * Certified LabVIEW Embedded Developer * Certified Professional Instructor * LabVIEW Champion * Code Janitor

Have you been nice to future you?
0 Kudos
Message 17 of 31
(3,297 Views)

kjeld wrote:

  • Test results as a Junit / XML file(s)

/christian

One more...

Unit Test Framework has the option of generating reports ATML/XML, but as everything else with UTF, it is not easy to find how to do it and not very intuitive.

The place to do it is at the project properties uner the Unit Test Framework category (where other goodies like changing the log test fiels, default file and test vector location, etc).

I think my biggest complaint about UTF is that is not "LabVIEW Developer" friendly. It breaks the flow when creating unit tests and makes things that should be easy difficult. However, it already has a lot of the features we need.

UTF project properties.png

For an opportunity to learn from experienced developers / entrepeneurs (Steve, Joerg, and Brian amongst them):
Check out DSH Pragmatic Software Development Workshop!

DQMH Lead Architect * DQMH Trusted Advisor * Certified LabVIEW Architect * Certified LabVIEW Embedded Developer * Certified Professional Instructor * LabVIEW Champion * Code Janitor

Have you been nice to future you?
0 Kudos
Message 18 of 31
(3,297 Views)

jon_mcbee wrote:

Unit testing code with hardware depenencies will end up being a common use case with LabVIEW.  Test doubles for hardware abstraction imply something about the way the software is written.  This is the thread of thinking that got me writing the unit test blog post, does the ability to write good unit tests require code that was written with testability in mind?

At some point the creator of a unit test tool will have to place the onus of writing testable code on the developer.  One thing I would like to understand is where that boundary exists.

At the CLA Summit in Austin we had this discussion, I have been thinking on writing a blog post about this subject as well. Now, I will just sit back and relax and wait to point people to your blog post once it comes out

The discussion at the CLA Summit started with the following argument:

"But now you are asking me to modify the existing code to fit the unit tests that I have to create"

First of all, a lot of this can be avoided if the unit tests are designed before or at the same time as the units they are testing are developed. You get extra points if you can have a different person write the unit test than the person who is writing the code. Given the size of LabVIEW teams, this is easier said than done.

Here is my analogy:

When hardware engineers are designing PCBs, they will have to keep in mind the test fixture that will be used and what test points they need. If this means that they will have to move traces, components around to make room for the test points, they will.

It is the exact same thing for us, if we really want to be able to test the structure of our code and increase our confidence that if we make a change we will know what other parts break... then we should be OK with making changes to our code to make room for this testing.

Create subVIs for small parts of code, use unbundle before going into a subVI and provide only the inputs needed for a subVI to make that subVI more testable, remove pop ups from low level code, etc...

All of this is a lot easier if developing is done with testability in mind since the beginning.

Excuse me for writing so long posts, hopefully people find my stream of consciousness useful. Unit testing is close to my heart and I am glad to see more people interested in it and seeing its benefits, this can only result in better tools and processes for all.

Regards,

Fab

For an opportunity to learn from experienced developers / entrepeneurs (Steve, Joerg, and Brian amongst them):
Check out DSH Pragmatic Software Development Workshop!

DQMH Lead Architect * DQMH Trusted Advisor * Certified LabVIEW Architect * Certified LabVIEW Embedded Developer * Certified Professional Instructor * LabVIEW Champion * Code Janitor

Have you been nice to future you?
Message 19 of 31
(3,297 Views)

Excuse me for writing so long posts, hopefully people find my stream of consciousness useful. Unit testing is close to my heart and I am glad to see more people interested in it and seeing its benefits, this can only result in better tools and processes for all.

I find your posts extremely useful!  Please, stream away!

Message 20 of 31
(3,297 Views)