Tracking down source of Termination

[mdwebster]

I have a test sequence that is terminating itself once out of every 20 to 30 runs during the middle of execution (not always in the same location) and there's no indication on the report of what is causing it (most steps and several subsequence calls have result recording disabled).

I am doing programmatic terminations in two places.  One is governed by a user interface if they hit abort test, and the other by an over current condition on an actuator we're monitoring.  Normally the overcurrent would be recorded on the test report prior to the terminate being called, but there are 3 places where I have subsequence result recording turned off on subsequences that contain those overcurent terminations since I'm just using them for process setup and not explicitly for testing.  Note I do not believe I'm getting overcurrent since that's a relatively rare state and it should show up in prior or subsequent testing but it doesn't get closer than ~30% of the requisite current to trigger in the steps where the results are recorded. 

Just to cover my bases though, in those subsequence calls where I have result recording disabled, I turned on the IgnoreTermination flag which should prevent them from terminating the calling sequence.  However, this did not solve my problem.

Other than explicit Terminate calls (which I assume edit>find of the word terminate should locate them all (further assuming I'm not calling it from within a LabVIEW VI)) or the operator pressing the terminate button or pressing ctrl+alt+F9, is there anything else that could trigger a sequence termination?

Thanks,

Mike

[JDragon]

Hi Mike,

This will be kind of tough to troubleshoot without more information. Specifically, we want to know which steps the termination occurs.

When you say that your test sequence is terminating is it going straight to the Clean Up or is the test sequence aborting?

You said this is happening in different locations. Can you narrow down those locations?

Does this only occur during action steps when calling a VI?

Can you see which test step executed last? Are any of the test steps skipped?

Can you enable result recording temporary and set break points were you think the issue may be happening? 

[mdwebster]

Hey JD,

I don't know where the termination is coming from.  I've done everything I know to prevent the only sources of programmatic termination from triggering (see IgnoreTermination flag).  I've seen it happen in two different subsequences (which both call the same sub-sub-sequences that contain programmatic terminations but which I've set the IgnoreTermination flag for).

This is a termination, not an abort, cleanup is run on the way out.

I can see the last test step that has result recording enabled, but those are something like 1 out of 10 steps.  Enabling result recording is going to blow up the test reports from 50 pages to 500+ and this is running in a production environment so that really isn't acceptable.  If it was happening with any regularity we could try it just for debugging but it only happens once or twice per week with the stand running 8 hours a day.

One of the big things I'm looking for is confirmation that terminations can only occur from sequencecontext.terminate, engine.terminateAll, or pressing the terminate button in the sequence editor.  If that is indeed true, then my only remaining theory is that IgnoreTermination doesn't work like I think it does.  I may try to create some StationGlobal debug flags and have post-step expressions to tell me if one of my programmatic terminations is triggering or not.  I really don't think they are given that they're not even close to triggering in other testing and I have IgnoreTermination turned on, but I don't know where else to look... 

Best regards,

Mike

[~jiggawax~]

I would throw down a SequenceFilePostStep callback and trigger a breakpoint on Execution.GetStates and look at the ExecutionTerminationStates.  I threw something together real quick.

At least you could narrow in on what step is causing it.

Hope this helps,

[mdwebster]

That's beautiful.  Thank you so much. 

I'll be back if/when the problem reoccurs (and it isn't something that leaves me with egg on my face that is ... )

[mdwebster]

Well, it looked beautiful.  Unfortunately, the only places where I'm doing programmatic termination are in post-actions which, apparently, triggers after the SequenceFilePostStep callback.  (If (locals.user_cancelled) terminate).

I assume that's it, the symptom at least is that the only things that trigger the popup are the cleanup steps.

I tried using Runstate.Caller.PreviousStep.Name, not knowing if it got the name of the step immediately prior to the first cleanup step or the last run step, but it throws a runtime error:

An error occurred evaluating the Message Expression:
Error in call to TestStand API member 'SequenceContext.PreviousStep'.
Unknown variable or property name 'PreviousStep'.
Error accessing item 'PreviousStep'.

Any helpful advice?

Thanks,
Mike

[~jiggawax~]

You could take a different approach.  I've attached how to do it.  The ideas is that you build a list of steps/sequences and then post them when you terminate.  Basically you are generating your own callstack.  You would need to add a FileGlobal called StepList that is an array of strings to your sequence file that you copy the SequenceFilePostStep to.

It injects them into element 0 so the list is reversed.  The item on top is the one that executed last.

Hope this helps.

[mdwebster]

This should work.  I need to use a StationGlobal I think because most of the routines I'm calling are in a separate sequence file (one support sequence file that rarely changes separated from test definition sequences that are mostly just defining order of testing and parameters), but that should still work.  My original ideas would have required much more work to implement.

Thanks again for the help,

Mike

[~jiggawax~]

I hadn't heard of this until today but my coworker just pointed out this little nifty tool (in a totally unrelated conversation):

https://decibel.ni.com/content/docs/DOC-26574

That might be very helpful for you in this case.