NI TestStand Idea Exchange

cancel
Showing results for 
Search instead for 
Did you mean: 
LVB

Add Result Recording Option "Enabled On Step Failure"

Status: New

When monitoring values within a loop in TestStand, it is often desired to only record step failure results.  It would be useful to have a "Result Recording Option" of "Enabled On Step Failure":

 

TestStand Idea Exchange - Enable Result Recording On Step Failure.png

 

This is possible through various means (SequenceFilePostResultListEntry callbacks and other custom code).  However, I believe this would simplify TestStand sequence development significantly.

CLA, CTA
6 Comments
AndyWatt
Member

I've implemented this as a mixture of various logging level controls and upward-percolating results reporting via a common cleanup step group handler - when a step fails, its' report is scanned by the cleanup handler and added to the result report for the whole sequence as an indication of why the step failed: eventually you get back up to top-level testcase code which then actually includes the result report, which can end up looking a little like a Java exception report but includes all failure reporting in the call stack for the sequence which failed. It works pretty well, and the logging levels for the levels of the framework I've implement allow me to log down to specific levels of complexity to cut down on log size (and create exceptions as well so specific sequences can be logged).

 

If this feature had been available it would have made my life massively easier, I could leave all critical steps as "log if failed" and not have to do the "bubble up"...

tyk007
Active Participant

This feature would be unbelievably useful for long running tests. Personally I think this should be standard.

TRJ
Member
Member

Using TestStand's Reporting or Database logging features, there are options to filter out passed step results from the reports/logged data. You can also tweak the filtering expression to log only the steps that Failed (in the option used below, both errors and failures will be logged).

 

Configure->Results Processing-> Report Options -> Result Filtering Expression -> Exclude Passed/Done/Skipped

OR

Configure ->Results Processing -> Database Options -> Logging/ Data Link Options-> Result Filtering Expression

 

1.jpg

 

 

 

You can generate reports that have only the Failed steps' results using these options.

 

Issues with this approach:

  • Will not work if you do not use TestStand's Reporting or Database Logging.
  • The filtering happens only when the report generation or database logging code is called at the end of the sequence run (unless you use the On-The-Fly option). The resultlist would collect all the result data irrespective of passed/failed status till it is handed off for processing.

I would also like to understand whether there still a need for a step level result recording option for Failed steps, given that these options exist?

mgustin
Member

Adding the Result Recording Option "Enabled On Step Failure" would be extremely useful.

 

The need for a setup level result recording option for Failed steps is because not all test steps are used to test the product. 

 

 Before running a test I have setup conditions which need to be tested but not recorded. I only want to record the setup condition test result if it fails. Then the test report shows the failing setup condition test(s) with the passing test results.

 

We use a sql database for data storage and use crystal reports to generate a report. The report data is pulled from the database thus insuring the data is in the database. It is very difficult to troubleshoot a system being tested when a report comes out "failed" but all the report data shows "passed". 

 

Recording all the "passing" setup condition test results clutters up the test report and database.

 

Bilby42
Member

Its been 4 years since this request was raised. I agree this feature would be extremely useful. I'm currently designing a burnin rig which will run for days, and in some cases only want to see the failed test results. 

 

Setting the global conditions as described by TRJ doesn't help as normally I want to see passed results, just not the ones I loop on 1000s of times over days. I'm assuming I'll need to write some custom result filtering function based on test names.

prc018100
Member

I've just started looking into the possibility of conditionally recording parameters in additional results based on whether the step passes or fails.  Adding step.result.status == "failed" in the condition field doesn't work because the status isn't updated before the results are logged (according to doug here: https://forums.ni.com/t5/NI-TestStand/Additional-Results-Condition-Not-Working-Correctly-For/td-p/30...).  Has anyone found another way to do this specifically for parameters?  Posting this here also to say that I think this option could be added to additional result parameters too!