Slow Unit Test Framework

[bokorpet]

Hi Vanessa,

for example: a simple VI, 5 test cases, around 2 seconds per test case.

It seems to me that the overhead is to switch from one test case to the next: if the block diagram of the VI under test is open, I can see for each test case the inputs/outputs associated with the test case. The test execution time remains the same if I close the block diagram. In this case I can see the "locked" symbol blinking (from test case to test case) of the affected project elements.

Is there an inherent runtime overhead of the test framework for reading in the next test case? Maybe there is a hard disk I/O access to the lvtest file every time a new test case is taken. This would be a very inefficient implementation though..

Any ideas?

Peter

[Vans]

Hi Peter,

unfortunately, you did not yet answer my question. A "simple VI" does not say anything about the execution time of your VI. I need numbers to verify your observation. So once again: How long does your VI take to execute once?

"if the block diagram of the VI under test is open, I can see for each test case the inputs/outputs" I hope you are talking about the front panel. Otherwise you are using the highlight mode. On my maschine I see the inputs and outputs of the test cases only on the front panel.

Please, take into accout that there is a lot of statistics done in the background. Two second per test case seem realistic to me.

Kind Regards,

Vanessa

[bokorpet]

Hi Vanessa,

yes, I'm also talking about the front panel; no I'm not in highlight mode 🙂

It must be some very heavy test stastics going on if it the instrumented code of this VI takes 2 seconds. I figure the native VI takes at most a few ms to execute..

I "only" need the PASSED/FAILED result and coverage information. I don't see it justified why it should take 1000x longer.

Regards,

Peter

[tyk007]

Posting the sample code that is causing you grief might help others replicate the issue. Otherwise it's pretty much shot-in-the-dark stuff.

[bokorpet]

Sure.

Here is a sample project. It's not the actual code but it will make it to explain my question.

The VI under test is a most primitive adder. I define 4 test cases in a test which takes around 3 seconds to execute. Again, I can see the inputs/outputs for each test case.

How can I make it faster?

Thanks,

Peter

[tyk007]

I was able to run your test cases total in around 4-5 seconds average over several runs. I would say there is an inherent delay involved in loading and unloading each test case; whether this is acceptable performance of not is up for debate. I noticed improved performance when using test vectors and a single test case - this dropped the test time to around 2-3 seconds.

If you are using OO then you might consider the free VI Tester from JKI as an alternative. This package is more closely aligned with the xUnit testing methodologies and you can manage how the tests are executed.

[Yamaeda]

What happens if you use DVRs for your classes?

/Y

[tyk007]

---

@tyk007 wrote:

If you are using OO then you might consider the free VI Tester from JKI as an alternative. This package is more closely aligned with the xUnit testing methodologies and you can manage how the tests are executed.

---

I ran a quick test using VI Tester from JKI in LabVIEW 2013. The response is a lot faster although more effort is needed to create and maintain tests; you will also have to rely a lot on community support. Just another option to consider if test time is really of a large importance.

[bokorpet]

@tyk007: I'm impressed, JKI seems to be a lot faster. Thanks!

Do you know by any chance if JKI can deliver some coverage statistics?

@Yamaeda: What do you exactly mean by using DVRs for my classes? We are bound by our development process to use LVOOP.

Peter

[tyk007]

JKI VI Tester does have some additional test creation over-head and a few anoyances when using the package with LabVIEW 2012 or 2013 (eg. template copying has bugs). I would recommend this for LabVIEW 2011 and earlier only until JKI have fixed this for later versions of LabVIEW to improve usability. Or better still, released to the community 

---

@Bokor wrote:

@Yamaeda: What do you exactly mean by using DVRs for my classes? We are bound by our development process to use LVOOP.

---

While I will leave Yamaeda to answer that query, I am unsure whether DVRs would have any significant impact on the unit test time given the simple example you posted. My guess is that there are simply inherent delays in between performing each test case. Additionally each time a test case is executed LabVIEW seems to load the entire call chain into memory and then unload after the test case is complete. The larger your object herirachy the worse that would get. At least it's automated though!

You could do a custom test case (UTF allows this) and write your own batch of sequential unit tests inside a single test case. This would reduce the burden in test time but increase the cost elsewhere (custom tests).

It would be good to get feedback on NI from this however I agree; the longer unit tests take to run the less likely they will be run.