Unit tests for RT target: InstaCoverage 2.1 is now available

[bokorpet]

Hey,

just wanted to let you know that InstaCoverage ("the unit testing tool for LabVIEW which is quite like UTF but much faster") has now been extended with a nice feature: unit tests can now also be executed on real-time targets (such as cRIO).

The latest version (called 2.1) is now available via the Tools Network page: http://sine.ni.com/nips/cds/view/p/lang/en/nid/216652

Regards,

Peter

[kosist90]

Hello Peter,

thanks a lot for sharing this tool with the community, it looks really interesting.

I was curious about it, and finally could do some tryouts.

Please, let me share some thoughts about the tool - and please, it is not criticism, or something like that, in no way. I was trying to summarize points, which - at my personal opinion - could be improved, or changed.

And on the first place, I am thinking about usability of the tool, and its "user-friendly" property. Because it tool is complicated, or requires many actions to do to run single test, then it is difficult to motivate team to use this tool, even if it gives some other benefits.

Looking forward when it'll be released as source-code - I guess, it'll be interesting to collaborate at it 😃

Points below are not so much sorted, but hope that it'll not be a problem.

1. Documentation of Setup/Teardown VIs (just some default one is enough, will be useful for unit tests newcomers) is missing. Would be nice to have there something like "This VI is used to set input data/init parameters for test", something like that...
2. Case structure in Setup/Teardown VIs is too small for code to add. Maybe, let it be by default bigger, so there possible to put some code without its manual resizing?
3. Possible to create unit test from private/protected VIs. But, generated VI will be broken. Maybe, makes sense just to warn user about it, and not generate VI -> because to run the test, developer would need to create wrapper. And, moreover, then such wrapper could be generated by InstaCoverage toolkit, and saved to library/class, to which VI-under-test belongs.
4. VIs are generated with long names. Check this out - I have method for actor's class. Thus, full VIs name will be: Library Name:Class Name:VI Name. Then, resulting test VIs names could be kind of this: "uiCfg_NodeEditor_Channel_uiCfg_NodeEditor_Channel_Launch uiCfg Core-teardown.vi". Which is not so much readable... But, I understand the reason of it - you just make sure, that VI's names will be unique.
5. Overall tests generation - I guess, it would be much better to store test VIs into libraries. At least, there could be done some main, default library -> and toolkit would refer to it first of all within the project (then, user don't need to select path to store test VIs at least). Another extreme would be for each test configuration generate separate library + save to unique folder. But, it will solve problem with unique, long VIs names. Because then, generated VIs will be in another namespace, and another folder - so no conflicts would occur.
6. Report generation - could be done as setting; it's a bit annoying to select this option again and again (whether generate reports, or no). The same could be for paths - once some folder is selected to store report, let the tool propose that folder again, and not the default one.
7. If Test Suite must contain at least 1 Test Case - then it is better to generate just 1 default Test Case label. Let developer to less work during test case creation. Personally, I guess the most ideal case is when test is generated, one goes to test_harness.vi, modifies condition to test - and test case is ready. For this there is no need to go to Config File Editor GUI, and edit it there, b/c it is some extra work. Of course, if more test cases are needed, with different configurations - then it is good to use Config File Editor GUI, but if it is not needed, then user don't need to be forced to use it. Now, there is anyway need to open it, b/c by default there are 2 test cases - and it is confusing...
8. Test re-run. This is something what I really miss... To run test again, one needs to close Test Suite Runner GUI, and then run tests again. It could be annoying... Personally, I have the following flow (for example, with VI Tester): open GUI for test run -> run tests -> tests fail -> go to VI-under-test -> correct it -> return to VI Tester GUI -> run test again.
The same is with Caraya - even if Caraya window is opened, one could re-run VIs with tests, and Caraya GUI will be updated.
Such behaviour is more user-friendly, as for me...
9. test_harness.vi - has VIs labels enabled by default. If VI name is long, then labels are overlapped. Maybe, it is possible to change it to not to show subVIs lables? After test_harness is generated, it has just 3 subVIs, and their meaning is visible based on icons.
10. Test result - contains just status, without any additional information. It reminds me Caraya. What I mean - is to have in report text input/output values, as VI Tester has by default. But, this is matter of approach, I guess with current implementation it'll be difficult to have. But, maybe you coudl think through, how to add some metadata to tests dynamically? Because now it is possible to add some description via configuration GUI, but again it requires some time. And if there will be some more rich API -> then it'll be possible to do dynamically, on-the-fly.
11. Uncovered Diagrams in results GUI -> maybe, it'll be nice to display there either default name as type of diagram (case structure, etc.), or if diagram has labels -> then label of the diagram. But, this topic could be much deeper, b/c toolkit could also list of pages which are uncovered...
12. If I add to VI-under-test, let's say, just empty case structure with boolean selector, toolkit also catches it as Uncovered Diagram. But, it's a bit not logical - b/c case structure even does not have some code inside + no output wires/nodes. In this case, maybe it makes sense to check, if structure page, which was not executed, contains any code. If no - then it doesn't need to be considered as Uncovered Diagram. I guess it'll make issues while testing class methods. By default, they are created with Case Structure and Error/No Error cases. But, if Error case page just passes value though - does it make sense to test via unit test? I guess no, b/c then there will be much more such test cases, but they do not give some added value.
But, if there is some code inside - and it means, that page does not simply pass data, but modifies it - then it makes sense to consider it as Uncovered Diagram, and warn user.
13. Config file editor: button "Open harness" could open block diagram, b/c I guess that front panel is not so interesting in most cases.
14. Test reports: html. It does not contain test description, and requirement description. But, this is something what could be displayed in the report...
15. Test reports: html. Name of configuration is done as hyperlink, but when click on it - nothing happens. Is there some reason, why it is done as hyperlink?
16. Test reports: xml. In case of failure, xml contains requirement text as "failure message". But seems, that failure message must describe why did test fail; but not describe when does that test pass.
17. Based on points 14-16: why does html and xml reports contain different information, please? I beleive that it has background, but as user I'd expect that any report type will give me in this case same information, but in different formatting.
18. Idea - to have API/hook, which will allow to generate test report in some other formats. CSV, JSON, etc. Could be handy in some cases.
19. Both GUIs have default context menu enabled. It looks weird a bit, b/c by simple "Reinitialize values to default" I could make path to VI empty - and red. And no way to undo that, just select path again =). But, this is just small detail...
20. Config file editor - it does not allow to select config file. It could be more user-friendly to add there button "Load config file", and open this window once, and edit - if needed - multiple cfg files in a row. Now, one needs to close window, go to project, and open new config file...
21. When run single test case (from cfg editor GUI), it does not discover Uncovered Diagrams. What is the logic behind it, please? Seems, that it is not described in the manual also...
22. Configuration file itself - now it is not so much clear its intention. I mean, that this file does not allow to set input/expected parameters for unit test, as NI UTF allows.
And now, interesting moment - I can specify any VI as VI-under-test. But does it make sense? Anyway, test itself is done within test_harness VI, there is called VI-under-test. So, to change my VI under test, I need physically to replace it.
What I mean - test_harness.vi works with AlgorithmA.vi. But, via configuration editor GUI, I set path to AlgorithmB.vi. Toolkit allows it, and then it will execute test for AlgorithmA.vi, but in test report will be specified AlgorithmB.vi as VI-under-test. Quite confusing...
And moreover, then it will include results for Uncovered Diagrams for AlgorithmB.vi, not AlgorithmA.vi.

Thank you very much,

Looking forward for your reply,

Sincerely, kosist90.

[bokorpet]

Dear Kosist,

thanks a lot for your feedback, we really appreciate your effort!

We'll look into your list in details and use it to plan improvements for future versions of InstaCoverage. Currently our main effort is to launch InstaCoverage for NXG, which we plan to keep very similar to (the existing) InstaCoverage for classic LabVIEW. The rationale is that users can have an aligned unit testing tool across different LabVIEW versions (classic LabVIEW and NXG).

We'll get back to you after we processed your list. One thing in advance: the main motivation of InstaCoverage was to be fast on the execution of unit tests in big projects while being able to measure test coverage too. In fact, InstaCoverage is up to 100x faster than UTF (e.g., 1h vs 1m) on avarage-sized projects (e.g., including several hundreds of VIs and dozens of unit tests). Also, we tried to come up with a intuitive test harness model for easy test suite/case design. 

Regards,

Peter

[bokorpet]

Hi everyone,

in case anybody is interested in getting news about the unit testing tool InstaCoverage, there is a newsletter which you can subscribe to here.

The upcoming new releases are:

• InstaCoverage Core (v0.2) for LabVIEW NXG and
• InstaCoverage (v2.2) for current gen' LabVIEW.

Regards,

Peter