Custom TestStand UI indicates Passed despite errors in execution

[jcarmody]

I have a test system that consists of TestStand sequences running in a LabVIEW user interface.  Most of the LabVIEW software is proprietary and I don't have the Block Diagram passwords so I don't have all the information I'd like.  I think I can describe the problem, however.

I witnessed a glitch that caused a run-time error during a test sequence, the operator accepted the 'retry' option and the test repeated successfully.  None of the tests in the sequence failed, but there was the one step that had the error.  Does it make sense that TestStand would consider the overall result to be a pass?  Is there a way to change this?

Thank you,

[Teds]

Hi

Here is my explanation.... I believe what you said does not sound like a glitch.... but expected behavior.

Go to Configure -> Station Options and select the Execution tab. 

Now look at the options for 'On Run-Time Error:' if it set to Show Dialog, it will prompt the user for an option to retry ( ignore and something else I believe? ) the error.  If it successfully passes on the retry it would essential 'ignore' the error and proceed like it NEVER happened.  That makes sense to me.  I take the idea of a Retry as you are attempting to do the operation again ( and again if it keeps failing )  and disregarding the previous result.  If you want to keep the first result, then don't allow them to do a retry.

If you don't want this to happen I would suggest to select a different option such as the ones below:

Run Cleanup

Ignore

Abort Immediately (No Cleanup)

We use 'Run Cleanup' as the default, in the theory that operators shouldn't be making decisions on whether to retry or proceed with errors that they essentially know nothing about.  Hence the status of the test on any error is always ERROR.

Thanks,

PH

[dug9000]

Are you saying that a retry was done on the step that had the error and then it passed? If so why would you not want the result to be passed? Make sure your steps don't have the "ignore run-time errors" setting checked on their run options. The default behavior is that if a step has an error then the sequence has an error. If you did a retry though that is perhaps why it wasn't an error anymore. Retry likely assumes you want to go with the retried result, rather than the previous result.

Hope this helps,

-Doug

[jcarmody]

Thanks PH & Doug.

After reading your responses I can see that the behavior makes sense.  Something that I forgot to mention (but adds to my confusion) is that the step that experienced the run-time error was an Action step which was not set to record its result, but did this time.  Is THAT expected?  Is it, perhaps, because a retry is considered an interactive execution because of the user interaction?

The "glitch" I mentioned was in my hardware; I wasn't suggesting that I found a bug in TestStand.  The end result of my trouble is that, while TestStand correctly identified the test as passing, the custom UI put "Failed" on the report.  I can't ship a product with a report indicating failure, even though it is a false-failure.

[Teds]

It sounds like the specific test ( sequence call ) that contained the retried indicates a PASS.... that is good. 

The report is still indicating an overall Test Status as 'FAILED" am I correct?  Is there any other failures in the report that link to a specific test or sequence?

Retrying the Action step still wouldn't give a result other than DONE, it would never give back a Pass/Fail result, unless it the status properties were modified specifically for that step.

That leads me to believe that you have another step and sequence that is failing somewhere.

I don't believe Interactive Mode is enabled unless you start the sequence such as 'Run Selected Steps using Single Pass", I don't believe it changes the 'RunState.Execution.InInteractiveMode' when running. That is a question for a more advanced TestStand guru.. than myself.....

Let me know what you find out...

PH

[jcarmody]

---

Teds wrote:

[...]

The report is still indicating an overall Test Status as 'FAILED" am I correct? 

---

You are correct, but the report is generated outside of TestStand and is not the issue. 

---

Retrying the Action step still wouldn't give a result other than DONE, it would never give back a Pass/Fail result, unless it the status properties were modified specifically for that step.

---

The initial execution of the Action step gave a result of "Error".  This, I believe, is what I need to address.  This step is not supposed to record its result, but it is.  The external reporting VI appears to require all "Passed" and "Done" results; this "Error" result is causing the report to indicate that the UUT failed.  I want to prevent the "Error" results from being reported, and the step Run Option "Do Not Record Results" doesn't appear to matter when an error occurs.

---

I don't believe Interactive Mode is enabled [...] 

---

That is what I'm most interested in.  I need to evaluate whether I can get away with turning off Record Results during interactive executions.  I think I probably can.  I won't have much of an opportunity to see if it works because I only had sixteen occurrences of this error in all of 2011.  Perhaps I can inject an error...

[jcarmody]

---

@jcarmody wrote:

[...] Perhaps I can inject an error...

---

I tested a sequence with one Action step and an OK/Cancel dialog where pressing Cancel injects an error.  I can't get it to exhibit the same behavior as my ATE.  Rats.