This is a call for any and all feedback regarding the LabVIEW Unit Test Framework Toolkit. Please, if you own or have evaluated this product, take a moment to share your candid thoughts.
Some questions to consider:
- why did you use or evaluate this tool? what problem were you hoping to solve?
- what motivated you to try the Unit Test Framework?
- what did you like most about the product?
- what was your overall impression of the tool?
- what didn't you like, or what do you wish it could do better?
And of course, this is a great opportunity to share any features or capabilities you would like to see added or changed.
Thanks in advance!
We would like to see UTF work on Linux. We still have not received our Linux version from NI for testing.
I am the one and only LabVIEW develloper here at getemed. I do create test systems for medical devices in a tighly regulated environment. The LV projects are quite large, up to ~1000 VIs and CTLs, and I have a lot of 'em and they share _some_ libs.
I have evaluated the UTFW-TK and was quite impressed. Good idea and good implementation. I have, however not checked the integration into our application lifetime management system, which is Polarion (see http://www.polarion.com/products/alm/index.php). As most of my 'projects' are still in LV711 (allthough I have all other LV versions since 4.x), I have decided to stay with what I do use at this time (individual test code created when develloping modules that in most cases create a small text file as test result).
What could have been done to improve the UTFW-TK was to have a quite generic integration link to and from ALM systems, probably based on XML-templates or, in case of polarion, MS-Word templates. Maybe you have it already and I have not found it, but than it should be made more recognizable.
We need our ALM SW and need to provide it as complete coverage on all related processes as possible in order to keep a reliable inpact analysis and traceability.
Just my € 0.02 from an evaluation of UTFW-TK that took place a few month ago.
Greetings from Germany!
I'm using UTF toolkit, among other labVIEW project, to review our SQLiteVIEW toolkit before each new release.
I like the way it's integrated in LabVIEW project.
One thing, I think you have to improve is the documentation. Not how to use UTF toolkit, it's well documented and not difficult to understand but the way to develop unit test is not easy to learn. In the first time I've use it, I've sometimes done more work on rebuilding my different tests than really work on adding features to my projects. It's really discouraging and can be a bad reason to not use UTF.
We can find a lot of document over the web about Unit Test concept but I've never find a document to apply this concept with UTF toolkit when I've started to use it.
Also, improve execution speed of UFT toolkit will be great.
I have used the UTF quite extensively to test code written in a controlled environment. I quite like having the tool integrated right into my LabVIEW project as it provides a good central location for handling the test runs.
There are two significant warts on the UTF though:
- It leaks VI references like a sieve. Test a simple VI using the UTF while running Desktop Execution Trace to see what I mean (this is current as of LV2010, don't know about 2011).
- The file format is too prone to corruption if edited by hand/in Excel. I personally would prefer XML, but even INI would be nice.
Neither of these issues have been showstoppers though - I still continue to use the UTF regularly.
We are currently using the UTF Toolkit for LabVIEW 2009 SP1 and have some things worth mentioning. We are obviously are using LabVIEW 2009 and use SVN as our repository. I would like to address Elijah's 'original questions to consider'.
Question: Why did we use/evaluate this tool and what problem(s) were we hoping to solve?
Answer: We deploy code on to RT targets all over the world and want need to eliminate deploying code that has obscure bugs that rarely show up.
Question: What motivated us to try the UTF?
Answer: The concept of unit testing is common among other software development languages and we were eager to incorporate unit testing within the LabVIEW project at the end of the development cycle and prior to integration into a real system. This will be a check that the developers have done their due dilligence and we can trust them when a developer says that "their code is done".
Question: What did we like the most about the UTF?
Answer: We like a) how it is incorporated into the project, b) either individual Unit Tests can be executed or all of them and c) the computation of code coverage.
Question: What was our overall impression of the UTF?
Answer: We have numerous projects that compise an entire set of code that is to go onto a RT target. It is a great start to validation of LabVIEW code within these projects at the developer level before it is integrated into larger systems.
Question: What didn't we like, or what do we wish the UTF could do better?
Answer: We have unit tests that are composed of numerous test cases.
a) The unit test properties have 4 categories - Configuration, Test Cases, Setup/Teardown and Advanced. The Comment and Requirement ID that resides in the Configuration category is for the entire unit test and not each test case. We need to have the ability to assign a Requirement ID to an individual test case and have an associated comment about the test case.
b) The progress bar that is displayed during the execution of a unit test does not update until the very end and think that this is actually a bug. The progress bar should be updated according to what test cast is executing related to how many test cases there are in total. A feature that would be nice is an indicator showing the test case number being executed.
c) There is currently no way to reorder the test cases. Lets say that we created 5 test cases within an unit test and there is a need for a new test case based on Test Case #2 but with simply some different values of the inputs. Test Case #2 is selected in the Test Properties window and we duplicate Test Case #2. It automatically will be assigned Test Case #6 now. We would like be able to reorder the test cases (similar to how you can edit the order of an enumeration) and put the new test case where it belongs.
d) This one is difficult for NI to help resolve without the actual code but we occasionally get the following error that shows up in the (HTML) report.
Test Case 18: Error
|1||VI Under Test||Possible reason(s): Test Case Data Error.|
We have no idea what is going on. Here is a link to the NI Discussion forum we created about this error:
These are some of the things we would like to see the UTF do better. In addition, the changes the 3 or 4 previous comments suggested would be beneficial to the UTF product. So obviously the UTF is a great start to helping with the overall validation process but it can be made better.
I just started evaluating the UTF yesterday and came across a problem. I've been searching for a solution, but haven't found anything yet.
I was testing inputs that would generate an error condition and saw that the test was always failing. In the VI under test I use 'Error Cluster From Error Code.vi' with 'show call chain? (False)' set to True. The issue is that the UTF would show up in the call chain with a unique identifier each time causing the comparison of the error description string to fail.
Good feedback. In the short term the best workaround is probably to change the cluster comparison type to "by element" and then set the source field to string length > 0. That will at least validate that the source is there although it won't be able to validate the actual call chain (which is obviously going to vary when using outside of UTF anyhow).
I completely agree with you that learning how to develop good unit tests is difficult and time consuming. On the other hand, I don't think it's realistic to expect NI to have all the answers on how to go about developing unit tests for the UTF. That's the kind of knowledge that develops over time as users experiment and share code and experiences with each other.
You may have already found this, but the best resource I've found to help me with the learning curve is the book xUnit Test Patterns. Granted, it is more applicable for those using LVOOP and JKI's VI Tester, but it may have some insights for traditional LV developers using UTF users as well.
To answer Elijah's question, I tried UTF a couple years ago when I was evaluating both the UTF and VIT. I thought VIT was more suitable for OOP developers while UTF seems more aligned with LV procedural programming. (As an aside, how about arranging a name swap with JKI? NI's unit tester is geared to testing single vis, so "VI Tester" is a more natural fit, and JKI's unit tester is more of a framework that you build on top of, making "Unit Test Framework" a more accurate name. [No, I don't expect this to actually happen.])
To answer Elijah's question, I tried UTF a couple years ago when I was evaluating both the UTF and VIT.
As an aside, it's difficult (if not impossible) to truly tie traceable test artifacts to third party tool results where the third party isn't ISO 9001 certified. That's a big deal for us when we choose toolkits to use on our regulated projects (which is a large chunk of what we do), and a big plus for NI's UTF.