Actor Framework Discussions

cancel
Showing results for 
Search instead for 
Did you mean: 

I'm thinking of formalizing verification of the Stop message in next rev of AF

A verified stop is when you tell an actor to stop but rather than stopping immediately and telling its nested actors to stop, it puts itself into a "shutting down" mode, tells its nested actors to stop, waits for all the Last Ack messages to come in and only then stops itself. In this sequence, the caller is guaranteed to be around through the whole shutdown sequence.

A few people have built a "verified stop" mechanism for their own work and more people have asked me about it, to the point that I think it might be worth adding for LV 2016 (too late for LV 2015 but certainly possible to have the community release sooner).

Here's my thought:

  1. Add a "Write Must Verify Stop.vi" to the base Actor class. As long as this is not turned on, the actor would behave exactly like today. If this is turned on, then the rest of this would kick in.
  2. In Stop.vi, check that flag. If the flag is off, do stop same as today. Otherwise, record that the actor is in "Shutting down" state.
  3. Call a "Shutdown Processes" subVI -- this is what would fire the Stop message off to nested actors. It would output an integer N of the total number of processes to wait for. Child actors could override this VI to add additional actors to wait for (i.e. nested actors that weren't registered when Launch Nested Actor was called) and to message other Actor Core loops to shut down.
  4. Modify Handle Message so that if the "shutting down" flag is set, the message is dropped instead of handled unless the message is a Last Ack message. <<< very unsure about just dropping the messages, but it seems like right idea. Alternate idea is to let actor continue to receive messages and then force actor author to put special "is shutting down" code in every handler VI. That seems icky.
  5. When a Last Ack message is handled, call "Decrement Stopping Count.vi" which drops the count noted in step 3. When that count hits zero, execute the rest of the Stop behavior.
  6. Introduce a Decrement Stopping Count Msg that can be sent directly to allow non-Actor processes to message that they are done ... needed for secondary Actor Core loops.

This is a very rough sketch of what the added feature would look like. Does it sound useful? Is it commonly needed enough to be worth adding to the framework? Even if it is useful enough, is this the way to do it? Or is there a better way?

Message 1 of 65
(14,396 Views)

One thing to note: new users of the AF are sometimes burned by the caller quitting before the nested is done. The verified stop would close that timing hole, so the nested actor would never get an error caused by early exit when trying to message its caller (baring the network going down for proxy callers or out of memory errors). That would be one argument for turning the verification on by default, but that would be a break in existing actor behavior unless we inject mutation. I'm open to suggestions that would let us turn this option on for new classes but not for existing classes.

Message 2 of 65
(6,064 Views)

Yes, it sounds very useful. I'm sure many of us have rolled something like this in our applications many times over.

This would no doubt work, however leaves me with the impression of being incomplete because the burden of creating a "Shutdown Process" override is left to the user. Without the override nothing works. Doesn't strike me as particularly helpful to the new users if solving their "timing hole" is truly an objective. Note I'm not criticizing the interface per se; it is valid and by some measure perhaps the best way of doing things, particularly if the goal is to keep the root Actor implementation as lightweight as possible. The useability from the perspective of a novice however is lacking in my opinion.

Instead I'd prefer a default implementation that does the book keeping. This will mean burdening the Actor object with extra state data.

Actor has a method to launch nested actors, during which a refnum is stored for sending messages to the nested actors. During shutdown, this list is used to relay the Stop message to those nested actors. One by one the refnums are removed from the list as the Last Acks are received (regardless of if they come from a relayed Stop or by the nested Actor stopping on its own). When the Last Ack is received which reduces the list to zero size, Stop behavior proceeds if the Actor is in shutdown mode.

Not to derail the discussion, this mechanism would make it trivial to implement chaining arbitrary messages to nested Actors (not just Stop).


There are timing subtleties at play however-- failing to relay a message can't be terminal since it would be possible for the nested to stop before the parent has had opportunity to process the corresponding Last Ack, but poor implementations of your original proposal could suffer the same fate. The potential for race conditions is a gaping hole here if the "Shutdown Process" implementation is left to the user, and I *know* how much thought you put into that when designing the actor framework as it currently stands.

Consider: My actor spawns six nested actors. So my Shutdown Process override will just say to wait for six Lask Acks. But what if one of those nested actors stopped prematurely before I was tracking Lask Ack messages? I'll only ever get five, so...deadlock. OK, better idea: I try to send messages to all six nested actors, but only return the number of successful relay attempts. Why doesn't this happen automatically? Oh right, the framework has no built in way of tracking launched nested actors. I guess I'll roll that too. But if I'm tracking, why don't I just use that data for deciding when to shutdown and forget about returning an integer? Data scope...

Back to the feedback at hand. I think dropping messages is a good thing if in shutdown mode, though maybe returning an error would be warranted? I don't like the idea of "is shutting down" code in every handler.

Haven't had time to fully think about non-Actor processes. Hope this isn't too much of a random stream of consciousness, it's late and I have to run.

m

0 Kudos
Message 3 of 65
(6,064 Views)

I admit I have never needed a verified stop. Up to now I send the stop to the actor, the stop core sends stop to all nested actors, and everything is fine. The only error I had was if trying to stop a nested actor that already stopped, which is not a problem. I'm not sure this feature should be part of AF and it certainly shouldn't be on by default.

What might be better is to prompt the user to close actors in stop core if there are actor queue refs in the private data of the actor, and definitely have a "Stop all running VIs" button in the IDE.

Thanks,

Danielle

"Wisdom comes from experience. Experience is often a result of lack of wisdom.”
― Terry Pratchett
0 Kudos
Message 4 of 65
(6,064 Views)

kegghead wrote:

This would no doubt work, however leaves me with the impression of being incomplete because the burden of creating a "Shutdown Process" override is left to the user.

I don't think they *have* to override it. I think if they've used Launch Nested Actor and put the system in charge of their nested actors (which is the default setting of the Boolean input on that VI), then we should be able to have a base class implementation of this. We already do that for Stop. The override is for when the user takes control of the shutdown or has non-Actor processes to worry about (i.e. if they need to halt their own Actor Core loops synchornously just like their nested actors).

0 Kudos
Message 5 of 65
(6,064 Views)

Ah I'm still working with 2013 where there is no such method. We're not so agile on taking up the latest versions.

I think the possibility for races with nested non-Actor processes would remain if things aren't tracked until after shutdown mode begins. Instead an "Attach Nested Process" method may be appropriate, where an abstracted transport mechanism is used such that the Actor can notify anything of the need to stop. The nested Actors could use the same mechanism only they would be using a Actor-specific implementation which sends a Stop message. Other implementations could fire off user events, notifiers, globals, whatever need be to get the signal through to the other end.

The nested processes would of course have the responsibility to send a Last Ack to the owning Actor, but this would be the case regardless of how things are tracked.

0 Kudos
Message 6 of 65
(6,064 Views)

I think that this would be a nice addition. I think how a program stops is just as important as how it starts. I find the current code a bit lacking. I also think with the addition of the storing of the the nested actors and stopping them by defualt (in LabVIEW 2014) means that you have half of the verified stop code in place.

My concern would be that the mixing of a verified stop and a regular stop could create confusion and lead to people having trouble during editting and testing with the project ending up locked (due to VIs left running which did not shutting down).

Another use case I am thinking of is a sequenced shutdown. For example a system with a shared error logger. You may want to shutdown from the top down but leave the data logger (which is 1 level from the top) running while telling other sibling actors to stop. How many use cases are special and shouldn't or can't be handled in a generic way? I believe this is the "Shutdown Processes" method.

At a certain point adding code to the base AF Actor can't handle every use case. So making sure you get enough without adding too much is a fine line to walk.

To I would suggest that a must have is the "Stop all running Actors" button in the IDE without closing LabVIEW.

A nice to have would be something like monitored actor which would allow you to see what actors are still running so that you can fix that code. Some visualization tools that show the tree of launched actors and what is active would be great.

Casey

Casey Lamers


Phoenix, LLC


casey.lamers@phoenixwi.com


CLA, LabVIEW Champion


Check Out the Software Engineering Processes, Architecture, and Design track at NIWeek. 2018 I guarantee you will learn things you can use daily! I will be presenting!

0 Kudos
Message 7 of 65
(6,064 Views)

CaseyLamers1 wrote:

I think that this would be a nice addition. I think how a program stops is just as important as how it starts.

I think everyone who has worked on AF design agrees with this. Indeed, managing "Stop" was *the* thing that lead to the creation of the Actor Framework in the first place. The other issues (deadlock/hang avoidance and resource management) were secondary to just trying to get a clean shutdown.

CaseyLamers1 wrote:

...
I find the current code a bit lacking.

...

My concern would be that the mixing of a verified stop and a regular stop could create confusion and lead to people having trouble during editting and testing with the project ending up locked (due to VIs left running which did not shutting down).

Your concern is to some degree why no verified Stop exists in the AF already. We looked at each actor as an independent entity, and left it open to the programmer to add an additional managment layer for those applications that needed it. But over time, I have seen that particular management layer come up more often, which is why I am exploring the option.

CaseyLamers1 wrote:

Another use case I am thinking of is a sequenced shutdown.

Noted. Will keep the use you described in mind for the rest of brainstorming.

0 Kudos
Message 8 of 65
(6,064 Views)

dsavir schrieb:

I admit I have never needed a verified stop.

Same with me, but I like the idea to have the possibility buit-in.

It also implies, that the Actor Object will have to feature a list of nested Actors by default, won't it?

0 Kudos
Message 9 of 65
(6,064 Views)

Oli_Wachno wrote:

It also implies, that the Actor Object will have to feature a list of nested Actors by default, won't it?

It already does. Upgrade to LV 2014. 🙂

0 Kudos
Message 10 of 65
(6,064 Views)