Unit Test Framework Toolkit Users: We want your feedback!

[Daklu]

LuI wrote:

BUT: One or the other example for non-trivial usage of any given tool would be considered VERY helpfull, especially when those examples are well documented. I'm not expecting explanation on WHY some test vectors where choosen, but why a special sequence of tests was chosen and what - if any - special comparisions or test criteria have been chosen.

This would help to understand what the devellopers had in mind how to use the tool. So one would work _with_ the tool and not _against_ it.

I understand what you are saying, but I respectfully disagree.  One of the problems with examples is the community in general view them as the NI *blessed* way of doing something.  I do agree many of the examples included with Labview are poorly written, but there is also an element of unreasonable expectations on the part of the community.  I can't tell you how many times I've talked to developers who complain that an application they built based on example code doesn't work right, is difficult to maintain, etc.

Any advanced example has design decisions baked into it that map back to specific (and often undocumented) project requirements.  The more advanced the example, the less chance it will apply to an arbitrary developer's project.  When developers copy the UTF examples and run into problems they will complain about how the example is wrong, inadequate, or whatever.

Learning how to use the UTF and learning how to develop a unit test strategy, write testable code, etc. are very different things.  NI can (and should) provide examples showing *how* to do certain things.  The *when* and *why* is not (in my experience) something that can easily be taught, and often it cannot be demonstrated in an example.

[rolfk]

Fully agree with you here, Daklu. Examples should be simple and show a specific feature. They never can be meant as an example how a complete application should be written, or they get to complex very quickly and still do not match with how 99% of the developers think things should be done anyways.

If you download the Windows SDK and look at those examples it is the same. They show about how to do a specific thing, but if you go about writing an app based on that code you end up with a very poorly written app.

[reidl]

Daklu wrote:

I'd be willing to give the UTF another try if someone can confirm the bug in 2009 causing LV to crash when trying to remove a unit test from the project has been fixed.

I verified that UTF in LV 2011 and LV 2012 does not crash when removing an .lvtest file from the project.

[mavimi]

One issue I came across, trying out the UTF, was that the output from VI under test is evaluated only after the teardown vi is run. Why would this become an issue?

Suppose you were testing a VI, that communicates with a database. In order not to open and close the database every time you need to query the database, you could construct the vi as a sort of functional global or action machine, where you have one action to open the database reference and store it in a shiftregister. Other actions to query the database using the stored database reference. And finally an action to close the database reference.

In order to test the database query actions, you need to have a setup vi that ensures that the vi's database reference is valid. Then run the query testcase.

And finally you would run a teardown vi to ensure that the database reference is closed.

But because teardown calls the 'vi under test' to close the reference, the outputs are altered. And because the outputs are evaluated after this step, it results in a failed test.

Is this intended behavior?

[reidl]

I'd like to understand this use case a little better.  Let's say you have a setup.vi, test.vi, and teardown.vi.  You also have a "dbmanager.vi" which is a functional global.

setup.vi would call dbmanager.vi with the action to open the database connection and put the ref on a shift register.

test.vi would call dbmanager.vi with the action to query the database ref and use it to perform database operations.

teardown.vi would call dbmanager.vi with the action to close the database connection.

What I am having trouble understanding is why you say "But because teardown calls the 'vi under test' to close the reference, the outputs are altered".  Why is teardown calling test.vi?  Shouldn't teardown.vi just be calling the dbmanager functional global?  At the point teardown.vi is run, UTF has already cached the values returned by test.vi and any action by teardown.vi could not affect those values.

If you could provide an example (perhaps accessing a file instead of a database) along with the UTF project, I could take a look.

Alternatively, you could also create a "User-Defined Test" which gives you complete flexibility over what VIs are run (setup, test, cleanup) and you determine the pass/fail conditions.

[mavimi]

Hi reidl

The thing is that dbmanager is the vi under test. Following your example:

1. setup.vi would call dbmanager.vi with the action to open the database connection and put the ref on a shift register.
2. the testcase would call dbmanager.vi with the action to query the database ref and use it to perform database operations. (The resulting data is on the outputs of dbmanager.vi)
3. ?
4. teardown.vi would call dbmanager.vi with the action to close the database connection. (now outputs of dbmanager.vi changes)
5. ?

If the output from dbmanager.vi, which is the vi under test, was evaluated at step 3, the testcase would pass. But it is evaluated at step 5, and at this point the output from dbmanager.vi is changed and the testcase fails.

I have constructed a small example that illustrates this, and will send it to you.

As promised: a small project illustrating the issue:

https://www.sugarsync.com/pf/D7341326_4490840_810798

The test named "unittest" calls setup.vi, that in turns calls db-query.vi, with the "Open DB" commands. Then the testcase runs, running db-query.vi with the Query command and at last the teardown.vi runs again calling db-query.vi but with the "Close DB" command. This alters the Query result so the test fails.

The test named "same test omitting teardown" does as the name suggest, and passes.

Well, maybe I just have misunderstood the purpose of setup and teardown vi's?

u

[reidl]

Thanks for providing the project.  I reproduced the issue and know why.  I will try to explain some of the reasoning behind why it behaves this way (and I had a wrong assumption on my part).

First, let's look at how UTF runs your setup, test VI, and teardown:  In order to support values being passed from the setup VI to the test VI, and from the test VI to the teardown VI, UTF will create a new VI and script the 3 VIs on its block diagram and do any necessary wiring between the VIs (in your case, there is nothing being passed because data is communicated via the functional global).

Second, UTF runs this scripted VI (which means it cannot stop between the test VI and teardown VI... it runs everything).  After execution, we obtain a VI reference to the VI under test (in your case, the functional global).  We then query the control values of the VI under test.  Unfortunately, as you pointed out, calling the VI under test from the teardown VI alters the front panel values of the VI under test.  My assumption that we could cache the values between running the VI under test and the teardown VI was incorrect because all 3 VIs are run from the scripted VI.

How can we get around this?  I would suggest having two VIs (ref manager and "query database").  The ref manager would be the functional global and its responsibility is simply opening/closing the database connection and returning the db reference.  The second VI would be the "query database" VI (VI under test).  In it, it would call the functional global to obtain the db reference, then perform the actual query.  The teardown VI would now no longer call the VI under test "query database".  It would be calling the functional global that closes the db reference.

[mavimi]

Thanks reidl, that does explain it.

Your work around would work, I think - but it arises the question - how do you then test the functional globals?

As I see it there is missing a way to call a vi to put data into it, and to call it again to test if the data is still there. You could call setup, and then test the vi, then the output would be valid. But in the database case, you need to close the reference using the teardown vi and results are lost.

My proposal:

Cache values after the testcase runs, before the teardown vi runs.

Or:

Include an option to keep the vi in memory in between test cases.

Then you could use fx test case 1 to put data into the vi. Test case 2 tests that the data is correct or as in my case that querying the database works. Test case 3 could then be used to close the reference.

Or:

Both 🙂

Of course I could create a user defined test, but I think it is pretty tedious, and I'd rather go with TestStand which has the options to control whether the vi is unloaded or kept in memory in between tests. Sadly I then manually would have to control code coverage 😞

I hope you take this into consideration, because this is what made me leave UTF and actually use TS instead.

[reidl]

One other solution to your particular problem right now... can the data (indicators) that are being validated be put on a shift register?  That way, the "Close database" action would just pass those values through instead of altering them, which is what your current example VI does.  I realize that this may not be ideal depending on how large that data is.

Of your proposed solutions, I would be in favor of seeing the values validated before the teardown VI runs.  Unfortunately, this is not a trivial fix because the scripted VI that runs the setup, VI under test, and cleanup is done in one atomic operation.  I can envision ways of fixing this but the changes may be far reaching.  We will file a CAR (corrective action request) regarding this behavior.

I would be against your second solution regarding keeping tests in memory.  The very nature of a unit test should not have dependencies on test cases being run before/after it or the order of tests being run.  There should be no residual state between test runs.

[mavimi]

Yes, you could put the data on a shift register of course. And I see your point against having dependecies between test cases.

But I'm glad you've decided to file a CAR, because from my perspective - it just is the correct way to do it - caching values (or validating them) immediately after the test case run.

So... thanks.