How can I detect that a step was forced to fail or forced to pass?

[tlaford]

Yes, it does what I am looking for, but not where I need to do it.  Changing a sequence file is not permitted, so I need to make the change in my C# application.  I have tried looking up "SequenceFilePostResultListEntry" but I have not found a TestStand event that corresponds to it.

[asbo]

Ah, I see - are you permitted to change the process model? Like I mentioned earlier, this could be implemented at that level as well.

I'm not having any epiphanies to accomplish this from the C# side - maybe you could use the fact that you have access to the Sequence object? Maybe something like ev.ctxt.Sequence.GetStepByName(ev.ctxt.PreviousStep.Name) will get you a reference to the step you can play with. Of course, make sure ev.ctxt.PreviousStep isn't null first 😉

Keep in mind that .ResultStatus and .Result.Status are different. From my experience, .ResultStatus only reflects in the UI and .Result.Status only reflects in the report (when you're changing them post-step, that is). In order for your Execution Viewer to update, you may need to refresh it as well.

[tlaford]

Unfortunately, Sequence is a ReadOnly property of the SequenceContext just like PreviousStep.

What object is the .Result.Status a property of?

[asbo]

Sequence itself is read-only, but is that necessarily inherited by the properties it exposes? Those should just be references to the actual objects in the TestStand engine.

Both properties I mentioned are sub-properties of a Step, though I couldn't tell you why both exist instead of just one. It's not listed in the help file as a property so it must be inherited from step type? Don't quote me on that.

[tlaford]

Because the sequence property is ReadOnly, any changes made to the step are not applied to the original.  I know, because I tried (see the sample code a few posts ago).

In C# I can only access Step.ResultStatus, there is no Step.Result.Status.

[Josh_W]

Just to make sure we're clear.  There is a property of the Step class of ResultStatus that is a string.  You can access this directly through the step interface.

However, a step is a PropertyObject (just like most everything else in TestStand), and so it can contain other property objects. 

One of the property objects that it contains is called "Result".  Using lookup strings, you could access the value you are looking for by the lookup string "Result.Status".  Keep in mind that when you type a TestStand expression (for instance into a watch expression) it will evaluate any lookup strings that you give it.

In C#, there is no inherent knowledge of lookup strings like there is in a TestStand expression box.  So you will need to use the TestStand API to access the data you are interested in.  If your step object is called 'myStep', then the following should return the result string:

myStep.AsPropertyObject().GetValString("Result.Status", 0);

At that point, you should be able to also use SetValString if you want to write to the property.

Hope this helps!

Tip: Lookup strings are NOT case-sensitive.

Message Edited by Josh W on 06-03-2010 02:22 PM

[tlaford]

Oh! Ok, I see where the .Status property comes from.

I have tried both of the code blocks below, and neither of them change the Step's Status when it is written out at the end of the execution (i.e., in the EndExecution event's Execution.ResultObject)

    private void axExecutionViewMgr_Trace(object sender, _ExecutionViewMgrEvents_TraceEvent ev)
    {
        string runMode = ev.ctxt.PreviousStep.GetRunModeEx(System.Type.Missing);
        if (runMode == RunModes.RunMode_ForceFail)
            ev.ctxt.PreviousStep.ResultStatus = "ForcedFail";
        else if (runMode == RunModes.RunMode_ForcePass)
            ev.ctxt.PreviousStep.ResultStatus = "ForcedPass";
    }

and

    private void axExecutionViewMgr_Trace(object sender, _ExecutionViewMgrEvents_TraceEvent ev)
    {
        string runMode = ev.ctxt.PreviousStep.GetRunModeEx(System.Type.Missing);
        if (runMode == RunModes.RunMode_ForceFail)
            ev.ctxt.PreviousStep.AsPropertyObject().SetValString("Result.Status", 0, "ForceFail");
        else if (runMode == RunModes.RunMode_ForcePass)
            ev.ctxt.PreviousStep.AsPropertyObject().SetValString("Result.Status", 0, "ForcePass");
    }

So I can only assume that the read only PreviousStep property really means read only.

[Josh_W]

tlaford,

Ah! Sorry, that's what I get for coming in the middle of a discussion; I didn't realize you were trying to modify the status when it is written out.

First things first:  Are you sure this is a good idea?  I notice that you're doing this in a trace message, which means that it's in your UI.  So keep in mind that this change will only happen when you run your sequence from your UI, not from a built-in UI, nor from the sequence editor.  Also, you're crossing a major boundary between UI and execution which is going to tightly couple your system. 

Just to be clear: you're making a change to TestStand's reporting (one major component) from your user interface (another major component) thereby coupling them together.  It's probably a better idea to change the reporting directly. I think it would probably be a better idea to make this change in your process model or in your reporting component.

So...on to the show.

The first thing you have to understand is how TestStand does reporting.  After each step is run, a Result container is created and appended to the Locals.ResultList of the sequence.  This ResultList is passed up the sequence call chain and eventually ends up in one of the reportgen_<reporting_type>.seq files.  I would recommend that you search ni.com and the TestStand reference manual for reporting/ modifying reporting/ etc. for more information.  The key is that ONLY information that has been put in the ResultList is available for the report to process.

The next important piece of information you need to know is the order of step action execution.  There's a great chart in the reference manual in chapter 3 (it used to be page 3-14, I don't know if it is still) that shows the order.  The important part is that the trace message that you're handing happens AFTER the Result container is created and added to the ResultList.  So you are changing the step's property, but because the Result container has already been created, it's not updated in the ResultList, and so it's not changed in the report.

The most unfortunate part of all of this for you is that the Result container does not have the information that you need about force pass/force fail.  That information only exists on the step.  So you need a way to get that information before the step goes out of scope.

Here are three ways to get the behavior that you want in order of what I consider best to worst:

1. If you can modify your process model, I would modify the ProcessModelPostResultListEntry callback sequence.  In that sequence, you have a parameter of the Result object, and you have a parameter of the Step object.  You could then query the step object for it's force pass/force fail status, and update the Parameters.Result.Status string.

2. There is an API method that you can call such that if a property exists on a step, you can include that property in the Result object.  The API method that you would want to call would be
   

   RunState.Execution.AddExtraResult("Step.TS.Mode","Runmode"),

   This would add a property Result.Runmode for each step.  Then you would have to modify your reportgen_<type>.seq file.  In that file, you would need to find where the step status is written (if you search for "Status", you'll probably find it).  At that point, you'd need to use the information in the Result object that we added to change the status string.  

3. If you cannot modify your process model and must do it in your UI, then you can query the step object the way you have been doing, but you will need to modify the Locals.ResultList container.  You will need to get the number of elements in ResultList (it's an array of containers), get the last element, and then modify the Locals.ResultList[<index of last element>].Status string.

Downsides:

• Keep in mind that (1) and (3) will impose a performance penalty since they happen after every step.
• Option (2) requires going into the Step.TS container, which is an internal data storage for a step.  These properties are not documented, and are subject to change.  So you may find that your code would not work (without an edit) if NI needs to make a change in future versions of TestStand.

Hope this helps!

[tlaford · Accepted solution]

Josh W. and Asbo,

Thanks for all the feedback. After additional design discussions, we have decided it is not important to identify that a specific step has been forced Pass or Fail. A Forced Fail is a failure and will need to be resolved before the UUT passes and the test is accepted. If a Forced Pass is encountered we will set a flag and mark the test pass as "conditional", and that will be enough to tell the operator that something stills needs attention.

        private void axExecutionViewMgr_Trace(object sender, _ExecutionViewMgrEvents_TraceEvent ev)
        {
            // If any step is skipped, toggle the flag
            if (ev.ctxt.PreviousStep.ResultStatus == StepProperties.ResultStatus_Skipped)
            {
                this.aStepWasSkipped = true;
            }

            // If a step was forced to pass or fail, toggle the flag 
            string runMode = ev.ctxt.PreviousStep.GetRunModeEx(System.Type.Missing);
            if (runMode == RunModes.RunMode_ForceFail)
            {
                this.aStepWasForced = true;
            }
            else if (runMode == RunModes.RunMode_ForcePass)
            {
                this.aStepWasForced = true;
            }
        }

And when reading the Results PropertyObject to create the report...

        if ((this.aStepWasSkipped || this.aStepWasForced) && (this.Status == StepProperties.ResultStatus_Passed))
        {
            this.Status = "CONDITIONAL";
        }