Suggestion: Get Rid of the "Actor-to-Caller" Queue

[justACS]

OK, how is Car supposed to find out that Traction Control has shut down?  Unless Car regularly polls TC for status information (undesirable in an event-driven architecture), TC has to publish that information somewhere, i.e. there must be some actual implementation in code for TC to publish the fact that it has shut down.  In Actor Framework, that mechanism is the owner's queue.

TC doesn't need to know anything about its owner.  It receives messages on its own queue (which can come from its owner, an actor it owns, or laterally), and publishes messages to its owner's queue, or other queues as specified.  The messages it sends can be supplied by the owner, so all TC needs to know is that it will send a message (stored as an attribute) in response to a message it receives.  It knows it has an owner, which is allowed in OO architectures, and it knows about any objects it created and must maintain (not only allowed, but required).  That's about as self-contained as you get.

[Daklu]

drjdpowell wrote:

Ah, but what if the government brings in regulation requiring all car subsystems send direct status messages to an automobile equivalent of a flight data recorder?

The requirement for "direct status messages" can only be satisfied by having the sensor outputs wired (physical wires, not block diagram wires) directly to the data recorder.  Once you insert software into the communication path it is no longer direct.  Whether or not your software subsystems meet the direct communication requirement depends entirely on how you define your subsystems.  (Personally I would reject a requirement like that because it defines implementation instead of the actual goal the requirement is intended to meet.)

drjdpowell wrote:

A simple tree is a great design, but what happens when you find a need to do something different?

Refactor to meet the new goals.  If you accept my argument that the requirement for "direct" communication is invalid, adding a data recorder is as simple as adding a new actor controlled by the car.  If you don't accept it, then you can insert publisher actors between the car and the subsystem who are responsible for distributing the messages to the car and the data recorder.  "But that's not direct!" you say?  Nonsense.  Traction Control Publisher and Engine Publisher can both be considered part their respective subsystems.

This kind of hierarchical control isn't a silver bullet to solve all problems.  It's a starting point.  It's simple enough to avoid getting bogged down implementing stuff you might never need, organized enough to provide clear division of responsibilities, and flexible enough to adapt to new requirements.

I'm not opposed to the observer pattern; I just find I don't need it very often.  Building an observer-based application adds complexity that might never be used.  Where's the value in that?

drjdpowell wrote:

It is NOT Traction Control's responsibility to tell anybody it has shut down.

(I had written out a response to this, then saw niACS addressed it in his reply, so yeah... what he said.)

drjdpowell wrote:

In fact, Traction Control shouldn't even know Car exists!

To put a slightly different spin on what niACS said, TC doesn't know Car exists.  It just knows there's something out there listening to its messages.

[drjdpowell]

Hi Daklu, I was thinking you might want to get involved.    And hi niACS.

Let me just sketch out how an Observer Pattern might be added to the Actor Framework.  It is not unlike Daklu's "Publishers" for Traction Control and Engine, except that rather than custom publishers for each component and application, there is a single generic publisher (called the "Observer Register") that is built into the parent Actor itself, and is configured for its specific application via messages from the Caller, Car.

The components would be:

1) An "Observer Register" object that would be added to the private data of "Actor.lvclass".

2) "Register Observer" Message classes whose "Do" methods call registration methods on the Observer Register.  For example, Car might package it's own queue inside a "Register for all Observables" message.  When sent to the Actor, this would cause the Observer Register to record the queue and then send all future updates of any observable to Car.  Other types of message might register for only specific messages or specific types of message.

3) A "Notify Observers" method.  Rather than send messages to the "Actor-to-Caller" or other queues, the child actors' methods will call "Notify Observers" on these same messages; this method will look up any processes registered to receive this specific message and send a copy of the message to each such queue.

Such a design is very easy to use.  Once it is initially tested, one never has to add to it with new child messages or publishers.  To create the network shown in Daklu's third example, one would merely create "Data Rec.", "Tr. Ctl." and "Engine", and then send two registration messages to each of the later two: one registering "Car" for all observables and one registering the data recorder.

>  OK, how is Car supposed to find out that Traction Control has shut down? ... i.e. there must be some actual implementation in code for TC to publish the fact that it has shut down.

In the design just described, the actor is publishing information via the observer register, but the internals of the actor doesn't actually have control, or knowledge, of who is subscribed for the information.  "Observer Register" is not provided with any methods that reveal information about who is observing the process.  "Traction Control" doesn't know about the structure of the communications it is participating in. 

>  TC doesn't need to know anything about its owner.  It receives messages on its own queue (which can come from its owner, an actor it owns, or laterally), and publishes messages to its owner's queue, or other queues as specified.

My version of TC doesn't know it has an owner, and doesn't know any other queues.  Yet the higher-level architecture could specify it as communicating with any number of queues.  And the architecture can be altered without either altering the Actor or building a custom "publisher" facade for it.

>  Building an observer-based application adds complexity that might never be used.  Where's the value in that?

That's just it; it isn't very complicated.  It's very simple.  Once you've built the Observer Register Class and a few Registration message classes, it involves little more than using "Notify Observers" in place of "Enqueue".  I've been starting to use such a design in my own version of actors, and I've found it actually simplifies the design of the actors themselves, yet is very powerful.

-- James

[drjdpowell]

Daklu wrote

>  To put a slightly different spin on what niACS said, TC doesn't know Car exists.  It just knows there's something out there listening to its messages.

What was your definition of an "Event Agent" you gave on LAVA?  "An event agent is an agent that defines its api without knowledge of any code external to it."  Knowing that there's something out there listening is a violation of that, even if you think it is a minor one.   My described actor with an observer registry is capable of being a true Event Agent.  

[drjdpowell]

Hello all,

I slept on this and would like to present a hopefully clearer idea of what I'm talking about.  Below is two diagrams of the Car example which distinguishes two related but separate relationships between the actors, that of information flow (or message passing) and that of expectations (other words that might fit here are control or requirements😞

Please note the unidirectional arrows in the Expectations diagram (in contrast to Daklu's diagrams).  "Car" has expectations on Traction Control: it expects it to exist, to respond to Car's commands, to notify Car of required information, and to notify Logger of information also.   No other actor has any expectations on any other actor in this example.  For example, that Logger receives information from TC is not a requirement of either Logger or TC; the entire information flow network is controlled and required by Car. 

Though the Information Flow diagram is more complex, the Expectations structure is a simple tree.  But a more important feature is that the branches are one-way; there are no bidirectional expectations. I would argue that it is bidirectional expectations that leads to problems, not a non-tree structure.  For example, the possibility of "lockup" when using synchronous request-reply communication is a result of two actors both requiring replies from the other. 

The Observer Pattern makes the difference between expectations and information flow more distinct.  An actor can participate in communication without expectations.  And there is a clear asymmetry in the two means of communication, sending commands to known objects and publishing information for unknown observers, that acts as a strong "assert" to the programmer (if I'm using that term correctly) to build their actors to have a unidirectional expectation relationship with their caller.  It's true that in the current Actor Framework, actors merely expect their caller to exist.  But this fact, plus the apparent symmetry of the Caller-Actor communication setup, makes it very easy to develop actors that have strong expectations of their callers, such as having the caller respond to the actor's commands. 

Given a clean, unidirectional network of expectations, I don't see any problem with a more complex network of information flow.  So I think the Actor Framework is asserting the wrong thing; it should assert unidirectional expectations, not a simple tree of bidirectional message passing.

-- James

[AristosQueue (NI)]

> The Observer Pattern makes the difference between expectations and information flow more distinct.

In the Actor Tree pattern (just coined the term), the presumption is that *somone* cares about every actor. An actor standing off by itself can be forgotten. But someone started that actor running. The actor may or may not know who that someone is, but that someone clearly cares that the actor start running. That someone is the protector, manager, and confidant of the actor. The actor can then be designed around telling that someone about changes in its own status.

My theory, on which the whole Actor Framework is predicated, is that the result of asserting this expectation into the system is a simplification of the actors as a collective whole. The threat of an "echo chamber" problem -- where a message bounces around, being repeated by a circular chain of listeners -- is virtually eliminated*, and if you do introduce additional non-tree communication paths into the system, you know where those are so as to guard against re-introduction of the echo chamber.

Moreover, it removes the ambiguity of order of message responses for every actor in the system. Under the plain Observer model, is too easy -- in my observation -- to spin up a nest of independent actors, have everyone register for the events they care about, and have no idea when a given event occurs who will respond to it first. The order of observers on the listener list is dependent upon the order in which everyone chose to register, and someone unregistering and reregistering moves their place on the list. With the message queues, messages go to one and only one place, and that actor handles the message. The Actor Tree pattern, assuming no shortcut communication paths are created, removes the handle order question, which becomes particularly important during shutdown operations -- making sure that there is a clean, definable shutdown order is critical for hardware, and having every part of the system hear about the stop message in an arbitrary order is often a problem.

So, that's my theory of actor communication. Might be false, but it's why the presumption of an actor caller exists, and it is why I push back on the standard Observer pattern.

===========

* It is possible to construct an echo chamber within the tree, but the tree structure encourages programmers to think in ways from the outset that create message types that do not cause the echo chamber. Rather than a generic "this happened" message bouncing all over, the messages themselves tend to be a "me to you explicitly" type of design that cannot be passed through the system, and so the next message in the chain becomes a more specific type that says "this happened over there", so the resulting system as a whole knows when to repeat and when not to repeat a message.

[drjdpowell]

AristosQueue wrote:

In the Actor Tree pattern...

That's good!  You should put this in "Using the Actor Framework", possibly in the introductory section.  It greatly aids understanding of the design of the framework, how it is intended to be used, and how it differes from descriptions of the "Actor Model" one might find on the internet.

Moreover, it removes the ambiguity of order of message responses for every actor in the system. Under the plain Observer model, is too easy -- in my observation -- to spin up a nest of independent actors, have everyone register for the events they care about, and have no idea when a given event occurs who will respond to it first. The order of observers on the listener list is dependent upon the order in which everyone chose to register, and someone unregistering and reregistering moves their place on the list.

That's true.  That is an advantage of the "Actor Tree pattern".  I'll have to think about that one.

-- James

[Daklu]

drjdpowell wrote:

Let me just sketch out how an Observer Pattern might be added to the Actor Framework... Such a design is very easy to use...

That's just it; it isn't very complicated. It's very simple. Once you've built the Observer Register Class and a few Registration message classes, it involves little more than using "Notify Observers" in place of "Enqueue".

It's easy to use once you've become familiar with the pattern and know how to use it. Adding code to the framework to inherently support observers *does* add complexity, and that complexity increases the learning curve for any other developers who pick up the code. I don't want to add that complexity unless there is a specific need for it.

drjdpowell wrote:

Knowing that there's something out there listening is a violation of that, even if you think it is a minor one.

Yep, I should have said, "An event agent is an agent that defines its api without specific knowledge of any code external to it." What I meant is the event agent defines the messages it sends rather than allowing the message names or content to be defined by the agent receiving the message. It is very similar to an observable, except event agents can be bound to their listeners at compile time and observables are bound to their listeners at run time. (I think of observables as a subset of event agents, so an event agent could support run time binding to the listener.)

Advantages: Code is more explicit, easier to read, and doesn't require as much supporting infrastructure.

Disadvantages: Some complex systems may require dynamic rebinding, which an event agent does not necessarily support.

drjdpowell wrote:

I slept on this and would like to present a hopefully clearer idea of what I'm talking about.

The diagrams I posted show message routing, not expectations. I don't think there is any difference between your diagrams and my second and third diagram.

drjdpowell wrote:

I would argue that it is bidirectional expectations that leads to problems, not a non-tree structure.

Too vague. *Any* two actors working together are going to have expectations on each other. Doesn't Logger expect the messages to be in a certain format? Don't all the subsystem actors expect to receive status changes from Car, even if they don't know about Car specifically? I don't think you can create a non-trivial program using only unidirectional expectations.

drjdpowell wrote:

And there is a clear asymmetry in the two means of communication, sending commands to known objects and publishing information for unknown observers, that acts as a strong "assert" to the programmer (if I'm using that term correctly) to build their actors to have a unidirectional expectation relationship with their caller.

I agree there is a big difference between command messages ("do something") and event messages ("something happened.") I think instead of "expectations" what you're really talking about is command flow--how do the command messages propogate through the application? Which actors serve as command originators and what commands do they originate? A program with crossing commands--multiple commanders issuing instructions to each other or to a common subordinate--is fertile ground for bugs. So is a pure observer-based program with no commands, as Stephen pointed out.

But what prevents cross-command situations from occurring is the programmer's awareness of them and diligence, not any given framework. It's just as easy to write crossing commands in an observer framework as it is in any other Labview code. If using observable objects helps you to keep the the command hierarchy in mind that's great. IMO it kind of obscures the important principle, especially when trying to teach the idea to others.

AristosQueue wrote:

In the Actor Tree pattern (just coined the term)...

The pattern can be applied outside of the Actor Framework, or even outside of LVOOP altogether. I think there's a lot of value in giving the principle a name, but can you choose one that can be more broadly used?

AristosQueue wrote:

...the presumption is that *somone* cares about every actor. An actor standing off by itself can be forgotten. But someone started that actor running. The actor may or may not know who that someone is, but that someone clearly cares that the actor start running. That someone is the protector, manager, and confidant of the actor. The actor can then be designed around telling that someone about changes in its own status.

My theory, on which the whole Actor Framework is predicated, is that the result of asserting this expectation into the system is a simplification of the actors as a collective whole.

[WARNING - Rambling thoughts follow]

I arrived at the same conclusion (through a much more circuitous and pothole filled route) and my messaging tree usually follows the hierarchy you espouse. However, lately I've been considering the message routing and command hierarchy as two different views of the application. Most of the time they are identical but I haven't convinced myself isolated actors and/or cross-tree messages should be completely avoided.  It's certainly much easier to verify any single actor is behaving correctly when they are not present.  Even without them understanding and verifying system behavior is still more difficult than I would like, especially when a message has to propogate through many actors on the way to its destination.

This idea isn't fully fleshed out, but I'm leaning towards defining the types of messages being sent between actors as a way to understand what should and shouldn't be done.  I've noticed three kinds of messages in my applications:

1.  Control messages.  These are the messages from a commanding actor to a subordinate actor telling it to do things that will cause the subordinate to change state, exit, etc.  A subordinate should always get control messages from the same commander.  These are usually verb-noun messages:  StartAcquisition, SaveData, etc.

2.  Status messages.  These are messages from a subordinate to the commander, updating the commander with status information.  Based on the subordinate's status message the commander may take some action, change its behavior, or it may do nothing at all.  They typically have a noun-verb format:  StateChanged, ErrorDetected, OkButtonPushed, etc.

3.  Data messages.  These are messages containing user data generated by a given actor.  They are also usually in noun-verb format:  JoystickPositionChanged, MotorSpeedChanged, etc.  The difference between data and status messages is data message receivers shouldn't change their behavior based on the message--they just update their internal copy of the data and use it from that point forward.

Here's the Car actor showing the different kinds of messages being sent.  (I forgot to add the data message lines from the publishers to the Car, but they should be there.)  Pure observer-based patterns have no control structure. Pure hierarchical messaging patterns don't necessarily simplify things.  I think the sweet spot is somewhere in the middle.

Another drawback of maintaining a strict command tree is the commanding actor has to stay running and in memory until the subordinate actor is done. That's not always necessary or desirable. For example, a large application may have a small application launcher component that displays a splash screen, loads files, allocates memory, and starts the main application loop. Once that's done there's no reason to keep it around waiting for the rest of the code to finish before allowing it to exit. Or an app may spin up a daemon that needs to remain running after the actor initially needing it has finished its task and exited.

My current working theory is an isolated actor (meaning it exists in a separate control tree but still exchanges messages with the rest of the app) is okay as long as it is smart enough to determine on its own when it should shut itself down instead of relying on a commander to tell it to do so.  Of course any actors communicating with the isolated actor need to be written in a way that deals with situations when the isolated actor isn't present.  Yes, this is an advanced use case that requires more code and creates more complexity, but I'm not quite ready to abandon the idea of isolated actors.

[drjdpowell]

Hi Daklu,

Before responding, let me formally change my original suggestion for the Actor Framework from "Get Rid of the Actor-to-Caller Queue" to "Explain the recommended multi-actor structure in the documentation".  I suggest modifying "Using the Actor Framework" to include an introductory section on the advantages of a tree structure of caller/callees, complete with a diagram of a tree.  Later on there could be a section on "Breaking the Pattern" with advice on how to do cross connections and the potential problems of such (with another diagram).  Part of the value of this section on exceptions is to reinforce the value of the basic rule of a tree structure. 

Any future suggestions by me on the advantages of my previously described use of the "Observer Pattern" should not be taken as an arguement against the current Actor Framework paradigm.  There are many ways to do things.

So anyway, Hi Daklu,

Such a design is very easy to use...

It's easy to use once you've become familiar with the pattern and know how to use it. Adding code to the framework to inherently support observers *does* add complexity, and that complexity increases the learning curve for any other developers who pick up the code. I don't want to add that complexity unless there is a specific need for it.

Isn't the "Observer Pattern", aka "Publish-Subscribe", a pretty standard pattern that any developer should know?  Admittedly, there is a learning curve to my "Observer Register" API, but it isn't that complicated.  And your alternatives involve a lot of extra boxes on the diagram and/or a lot forwarding of messages through lots of VIs and loops.  I must say that all the code of yours that I have ever looked at has been perfectly clear, but it seems to me that just the increased number of components is added complexity in and of itself.  And even if it is clearer, I'm not sure it is worth the tradeoff of all the extra work of all the "publishers" and "proxies" and having to trace messages up and down through several layers when chasing bugs.  Particularly as learning my "Observer Register" API is a one-time thing, after which my (fewer in number) diagram boxes are about as simple as yours.

OK, I may be deluded about my API being not that complicated, but I still think that, once learned, it has a significant payoff. 

Daklu wrote:

Yep, I should have said, "An event agent is an agent that defines its api without specific knowledge of any code external to it."

I think knowledge about the higher-level program structure (such as AQ's "me to you explicitly" messages) counts as specific knowledge.  Your code knows it is sending messages to one thing.

drjdpowell wrote:

I would argue that it is bidirectional expectations that leads to problems, not a non-tree structure.

Too vague. *Any* two actors working together are going to have expectations on each other. Doesn't Logger expect the messages to be in a certain format? Don't all the subsystem actors expect to receive status changes from Car, even if they don't know about Car specifically? I don't think you can create a non-trivial program using only unidirectional expectations.

Well, no, my Loggers don't require the actors being logged send specifically formatted messages.  And I agree with your later definition of status messages being "from a subordinate to the commander", so "no" I wouldn't have subsystems expect status updates from Car.  The subsystems supply an "API" of commands or messages that they accept, and other messages that they publish.  I guess you could say the expect to be told what to do by somebody, and that they will be told via the messages they define. 

I agree there is a big difference between command messages ("do something") and event messages ("something happened.") I think instead of "expectations" what you're really talking about is command flow--how do the command messages propagate through the application? Which actors serve as command originators and what commands do they originate? A program with crossing commands--multiple commanders issuing instructions to each other or to a common subordinate--is fertile ground for bugs. So is a pure observer-based program with no commands, as Stephen pointed out

A "pure observer based program with no commands"?!?  Who's talking about that?  I certainly wasn't suggesting no commands.  Commands are sent; events are observed.

Daklu wrote:

But what prevents cross-command situations from occurring is the programmer's awareness of them and diligence, not any given framework. It's just as easy to write crossing commands in an observer framework as it is in any other Labview code. If using observable objects helps you to keep the the command hierarchy in mind that's great. IMO it kind of obscures the important principle, especially when trying to teach the idea to others.

True, but if we're talking about "asserts" here, then "Publishing" suggests information provided, not commands issued.  Similarly, a programmer could make an arbitrarily complex non-tree with the Actor Framework.  Yet, the frameworks "asserts" a simple tree strongly enough to prompt my initial (anti-tree) post.

My division of messages is very similar, Commands and then two types of published information by the Observer Register: state notifications and event notifications. 

[Daklu]

drjdpowell wrote:

Isn't the "Observer Pattern", aka "Publish-Subscribe", a pretty standard pattern...

Yes.

drjdpowell wrote:

...that any developer should know?

No.  Well... yes they "should" probably learn it, but I think it's a mistake to assume they already do know it.  OOP ideas and principles are still very new to the Labview community.  Introducing someone to the observer pattern isn't something you want to do when they're trying to implement a time critical bug fix.  In a couple years I might agree a Labview professional should already know the pattern, but not yet.

drjdpowell wrote:

And even if it is clearer, I'm not sure it is worth the tradeoff of all the extra work of all the "publishers" and "proxies" and having to trace messages up and down through several layers when chasing bugs.

As I indicated earlier, I'm not completely happy with tracing through several layers of code to make sure a message gets to its final destination.  Currently I usually resort to integration testing for verification but I'd prefer an easier way to verify via inspection.  There may not be a practical way to do that.  I haven't decided yet...

However, inserting a publisher between a message sender and message receivers isn't a lot of extra work.  On the whole I think it entails less work.  The core functionality can be placed in a parent class with each specific publisher being a child of it.  So from that perspective you're writing the same amount of code, it's just a matter of where you're putting it.

Furthermore, direct connections between two actors require they have compatible interfaces.  For an actor to be easily reusable it has to define its own inputs and outputs, not define them for interacting with a specific actor.  If you directly connect all your actors you have to make sure each actor's interface is compatible with all the other actor's interfaces.  That's fairly manageable if all the actors are application specific, but much, much, more difficult if you have a reusable actor library.

What do you do when you want to connect two actors with incompatible interfaces?  You're going to have to write application-specific glue code to do the api translation--often an adapter or a mediator.  Why give a reusable actor the ability to send messages directly to an arbitrary number of receivers when the messages are going to have to go through a custom translation layer anyway?

drjdpowell wrote:

I think knowledge about the higher-level program structure (such as AQ's "me to you explicitly" messages) counts as specific knowledge.  Your code knows it is sending messages to one thing.

I disagree.  Growing up I used to sometimes wonder what the girl I would eventually marry was doing at that moment.  I knew there was "one person" who was out there, but I didn't know anything about that person other than she was female.  I wouldn't consider that specific knowledge.

The specific knowledge I was referring to are the details that make the receiver unique--in this case its api.  The commander has specific knowledge of the subordinate's api; the subordinate does not have specific knowledge of the commander's api.  It's the commanding actor's responsibility to know the subordinate's api and make sure they can communicate.  With the tree pattern the "api dependency" is clearly defined for the whole project.  In the observer pattern it is the receiver's responsibility to match api's with the sender.  When everyone is talking to everyone it's not always possible for the receiver to change its api when an incompatibility is found.  (Like if the receiver is part of the reuse library.)

drjdpowell wrote:

A "pure observer based program with no commands"?!?  Who's talking about that?  I certainly wasn't suggesting no commands.  Commands are sent; events are observed.

The observer pattern is based on events.  A subject publishes an event and all the observers choose to react or not react to it.  There are no commands because a command requires specific knowledge of the observer's api, which the subject doesn't have.  The observer pattern (in my experience) is entirely reactive, not proactive.  No actor tells another actor what to do.  If your subjects are sending commands to observers you aren't following the observer pattern.

On the whole I agree with you that there are times when it makes sense to allow cross-tree messages.  But imo those are exceptional cases to be dealt with on a case by case basis, not something that should be built into every actor.  If you want an Actor-Publisher you could probably subclass the Actor object and create one.