Race condition preventing AF from reporting Actor errors?

[David_Staab]

I have an actor that launches some callee actors. With the DETT, I'm consistently seeing that the caller actor shuts down its message handler (by executing Stop Core.vi) before its callees have shut down and sent their Last Ack Msgs to it. Since the Last Ack Msg is the only way for a callee to send its final error state up to its caller, these errors are getting dropped before I'm notified of them.

Is there a workaround to this behavior? A better design approach than sending Stop Msg to the callees from the caller's Stop Core.vi?

[komorbela]

I guess if you want to handle all the callees you cannot just send stop to caller. If you send a stop message to the caller you have to take the risk to miss some messages.

I would rather have a separate procedure in the caller that tries to stop all the callees, listens to the Last Ack messages with a timeout, handles all the Last Ack messages from the callees that sent it, and logs the one that didn't (as an error, or possible source of a bug). After caller handled all it's callees, it can send Stop to itself and stop.

[David_Staab]

I suspected as much. I have the framework of a synchronous "Request Shutdown Msg" in my custom extension of the AF, but I haven't put much thought or testing into it yet. I'd definitely prefer to see something of the sort built into AF itself...catching all unhandled errors seems like a critical design requirement for a general-purpose framework. (Of course, I don't know whether the general theory of actor-oriented design has a conflicting requirement that  trumps this one.)

[AristosQueue (NI)]

David: Put your errors into a message class that overrides "Drop Message Core.vi". In that implementation, put something that logs the unhandled errors. This is the function that is called when a message is sent to a caller that exists at the time of enqueue but that quits before actually dequing that message.

[David_Staab]

Yeah, I was looking for Last Ask Msg's override of "Drop Msg Core", but it's not there. Your recommendation looks like the best way to extend the framework without overwriting it.

I'm not completely sure of how you suggest I do this, though. Errors come from three places:

1. The output of Do.vi, where they are guaranteed to be caught by Handle Error.vi.
2. A received Last Ack Msg, where they are guaranteed to be caught by Handle Last Ack Core.vi (and, if propagated, again by Handle Error Core.vi)
3. The output of Actor Core.vi, where they are bundled into Last Ack Msg by the framework itself and sent to the caller.

In #3, it can't be known whether the caller sees the message and handles/reports the errors. If it does, handling them in the callee that threw the error might duplicate an error report (presuming that "handling" = "any action that doesn't clear the error"). Not handling them in the callee might result in dropping/ignoring them.

It seems like either (1) I need to put ALL errors output from AC.vi into a custom "Callee Error" msg and then clear them, or (2) the framework should be modified to include "Drop Msg Core.vi" extensibility for the Last Ack Msg. Is that correct?

[AristosQueue (NI)]

So I have given some thought to your original question.

My answer is that you don't want to be sending the Stop message in the first place. Stop is the built-in Actor Framework message that tells an actor to STOP. You're done. Do nothing more. But what you want is to have an actor that gets a message and then keeps doing stuff. You want to send a custom Begin Shutdown message, or something like that. That message will send the Stop message to all the nested actors AND will put a state flag in your actor that all of its methods check... if any further messages are received *other* than Last Ack messages, silently ignore them or send a message to the caller saying "Already shutting down." In your Handle Last Ack, store off the errors from each child and decrement a counter in your actor object until you have the Last Ack from all the children, then just return the Merge Error of that array of errors merged with error code 43, which will cause your actor to exit one way or the other.

[David_Staab]

Wow, that is heavy duty. If this is something I need to do in every actor to guarantee a graceful shutdown, shouldn't it be built into the framework somehow? Or templatized to guarantee good design?

[justACS]

Yes, but you may be able to use inheritance to minimize the work.  Create a child of Actor with that functionality, and inherit from that child instead of from Actor directly.  You could also do something with State Pattern Actor, where each relevant actor has a "shutting down" state.

[lukepike]

This might not be something you'd want to do, but it might help if you edit the Actor and Last Ack classes to allow a custom Last Ack message to be set when the actor is launched and used when the actor shuts down. This is something I've been thinking about doing because having a bunch of cases in Handle Last Ack Core is pretty messy and hard to maintain. I'd much rather have a series of Handle [some actor] Last Ack methods that get called automatically by their respective Last Ack child classes than have to cast multiple times through many case structures. It looks like it would be easy to do, just a Last Ack class in the actor's private data, an accessor to write, and a class input to Send Last Ack.vi so it uses the right class.

[AristosQueue (NI)]

David_Staab wrote:

Wow, that is heavy duty. If this is something I need to do in every actor to guarantee a graceful shutdown, shouldn't it be built into the framework somehow? Or templatized to guarantee good design?

Maybe, but so far, I haven't needed it or seen a common need for it. Generally, normal shutdown for the root actor is a state it enters into ... like a user hitting the Quit button on the UI. It never gets a Stop message at all, and it sends Stop to its nested actors, and they reply back with the Last Ack. Either that or during shutdown, the whole point is to be async and no one cares about the errors... each one is in charge of itself and should do whatever it takes to shutdown. Until now, no one has proposed needing this kind of shutdown from an arbitrary Stop message.

It isn't the sort of thing that should be in Actor.lvclass because it isn't something that leaf level actors need. So perhaps you should create a child of Actor for "Mid Level Actor.lvclass" and see how far you can go with making it reusable.

I wouldn't call it "heavy duty"... it's the work of an hour or so. It would be significantly easier to implement if you pull down the slightly tweaked AF that I posted for the remote actors. I added one method to the base actor class, and that one method would give you the intercept point for dropping all messages after Begin Shutdown was received, so you don't have to put that check in every method.