Unit Testing Group
Announcing Caraya – A New Take On LabVIEW Unit Testing
How many times you have had good intentions to write unit tests for your LabVIEW project but when the time came you had more pressing fires to put out? Or how often have you created a small debug VI to test your software but never created a unit test to cover the same functionality?
I'm excited to announce JKI's new experimental assertion and unit testing framework Caraya. It takes a whole new approach to unit testing; your VI is your test. Caraya allows you to convert your manual test VIs you use for debugging into unit test cases with nearly no effort. Even running the tests is easy, just run the test case VI. Caraya significantly lowers the barrier for developers to write unit tests leading into improved overall code quality in real-world projects where developers don’t always have the luxury to write unit test cases first.
I'd love to hear your feedback. What works well and what could be improved. Please try it out and let me know.
Visit the Caraya GitHub page to get Caraya and learn more.
This is a pretty good tool you has created, it has potentials
I think it would be even better if it was possible to right click on in the project explorer on the VI you has created the unit test for and run the associated unit test. In this way it is easy for others to see if there is a unit test created. This is an issue we are facing in our project because it is a large project with multiple developers.
An other issue. You make a test all VI for running all the unit tests. Here it would be cool if there is way for searching a project for unit tests and run them, like a button at the to of the project explorer. This would be be more fail safe when automating the test from Jenkins or TeamCity, maybe byhaving an API for the tool. The developer is not always adding the single unit test to the all VI
Is it possible to get a report like NUnit or JUnit xml from the test?
I have a question for your video. Why are you applying the the assert test in paralle and not in line?
kljeld
Thank you for your great feedback. Your questions get directly into the heart of the philosophy why we created Caraya and what makes Caraya different. Unfortunately Caraya is designed to be something very different that what you may be looking for. But you may still fall in love with the simplicity of it even though it does not and maybe even cannot have some of the functionality you are requesting.
JKI's VI Tester, another unit test framework from JKI, is designed to work the more like the way you describe. It allows you to run all the tests for a project as you suggest. I recommend using VI Tester, it is a great and proven unit test framework and is a really good fit for the things you're requesting.
With Caraya we intentionally took a very different approach: a test case is a VI, running the VI runs the test case, you can compose test hierarchies by using test cases as subVIs. These are the fundamental design principles of Caraya that makes Caraya special. But also creates some limitations on the aspects you were requesting.
I think it would be even better if it was possible to right click on in the project explorer on the VI you has created the unit test for and run the associated unit test. In this way it is easy for others to see if there is a unit test created. This is an issue we are facing in our project because it is a large project with multiple developers.
As there is absolutely nothing special about Caraya test VIs there cannot be anything special about running them from the project. They are just VIs and we want to keep them this way to keep Caraya simple. LabVIEW already provides a very good support for running VIs. If someone would create a separate tool for running VIs directly from a project explorer, that would be great but there is nothing special about Caraya test VIs in this perspective and hence such a functionality really shouldn't be part of Caraya.
An other issue. You make a test all VI for running all the unit tests. Here it would be cool if there is way for searching a project for unit tests and run them, like a button at the to of the project explorer. This would be be more fail safe when automating the test from Jenkins or TeamCity, maybe byhaving an API for the tool. The developer is not always adding the single unit test to the all VI
For the same reason there cannot be a run my application button in the project explorer, there cannot be run my test button in the project either. There is simply no way of knowing what to run. As LabVIEW doesn't know what is your application and how to run it, LabVIEW doesn't also know what is your top level test VI and how to run it. This is a design choice we made with Caraya that makes Caraya very easy and very powerful tool for unit testing. But because everything is code it makes it more difficult to automatically discover all the tests, reconstruct the test hierarchies properly and execute them. I don't see this functionality be applicable to Caraya. Of course someone could write a heuristic code analysis tool that traverses all VIs in your project to determine which ones are tests and which ones are top level tests that need to be called and then calling all the top level VIs but then you kind of loose everything the speed that makes Caraya special.
What could be added, in principle, is a build spec for running the unit tests similar to a build spec for building and EXE, where you define your top level test case. But again your tests are code and your composition of tests into hierarchies is code. It is up to the developer to specify in code how the tests would need to be run. This provides great flexibility for the user but does not work well for the use case of searching through the project on what to run.
Is it possible to get a report like NUnit or JUnit xml from the test?
Right now we only support text reports but we are planning on adding a report provider interface allowing users to customize reports. Caraya allows infinitely deep and complex test hierarchies that may be something that is not supported by NUnit or JUnit xml report formats. If that were the case we would need to extend the report format to work with Caraya making it potentially incompatible with report viewing tools when Caraya would be used to run complex test hierarchies.
I have a question for your video. Why are you applying the the assert test in paralle and not in line?
This is a great question. As the tests are code, we can choose in code weather to run the tests in parallel or sequentially. When ran in parallel, all the tests are independent from one another. When ran sequentially, the previous tests can impact the tests following them. Especially if a previous test fails and returns an error then the following tests are not executed following normal LabVIEW execution pattern (however Caraya still reports that the tests received an error as the input). The reason I had the tests in parallel is because I wanted all the tests run independently of one another and the outcome of one test not impact the other test. This may not always be the case and you as a developer need to choose what would make most sence for what you are testing. For example if you are testing database operations and you cannot connect to the database you may not want to run all the database tests so it's okay to have the database test VIs follow the set-up VI connecting to the database.
As a rule of thumb you most likely want your tests to run in parallel unless there is a dependency between two tests (such as "write key-value pair" test executing before "read value by key" test).
Tomi
