NI needs to teach more actor-oriented programming principles

[Todd_Lesher]

AristosQueue wrote:

... imagine how much work would be saved if every piece of hardware came with an already rolled actor framework API ready for communicating with it ...

I'd like to see just one of those - and then just one calling actor. Eli's "Configure, Acquire, Measure" is both more complicated and simpler than I need. And it puts all DAQ in one method, as opposed to exposing an API via messaging.

[wirebirdlabs]

@Daklu wrote:

@JackDunaway wrote:
Guess I'm just saying that AOD principles don't have to necessarily originate as NI curriculum;

You are correct that they don't necessarily have to originate from NI.  However, over the past several years I have learned it is extremely difficult to gain significant traction in the community with any messages...

If a Community Group were created "Actor-Oriented Programming Enthusiasts" on 2011-07-28 instead of "Actor Framework", the message you just wrote would end up in the subscription inbox of 13 members rather than 284+ members. I think we would both agree that understanding principles and tenets of AOD is more valuable than understanding the syntax of AF -- but AF has been an important, concrete vehicle for raising awareness of important software engineering principles. Even though I have not and may never develop production code with AF, studying it and reading conversations on this group have added significant value to my designs. And I have a hunch that the pain points felt by trying to scale AF applications will result in R&D efforts with benefits far beyond just AF performance.

@Daklu wrote:

NI has spent too much time pushing the Actor Framework and nowhere near enough time teaching people the principles of actor-oriented programming.

All i'm trying to point out is an argument to this one statement -- I see NI pushing AF as probably more valuable than pushing AOD. This is not a value judgement on AF or AOD. It's just an observation that efforts pushing AF have generated positive effects that probably could not have been achieved by pushing AOD.

[Daklu]

(Preface: Once again let me clarify that my posts are not intended to disparage the AF or the time and effort Stephen, Allen, and Eli have put into it. I think it is a good framework and I agree with Jack that the community as a whole is better off with the increased visibility of AOP.)

"This is the person I had in mind when I wrote the main documentation for AF... They had no idea how to build the architecture they needed for their project."

Right, I understand that and I think you did a remarkably good job of creating a framework for those users. It can't completely prevent users from making the typical QSM mistakes, but it certainly guides them in the right direction.

I think our disagreement is rooted in a slightly different goals. You want to give users struggling to stay afloat a life preserver so they don't drown. That's great. A person in danger of drowning needs a life preserver. I want to teach users to swim so they can navigate the water on their own, and more importantly, decide if they should jump in the lake at all. There is a need for both approaches. Thus far the messaging coming out of NI has been, "Here's your life preserver. This is all you need." There has not even been an acknowledgement that users would benefit by learning how to swim.

When people venture into the world of concurrent programming, there are certain concepts and principles they need to understand if they want to make the best choices for their particular situation. Concurrent programming is inherently a more complex beast than single-threaded programming. There are issues people need to be aware of if they hope to avoid making costly mistakes. Labview and AOD makes concurrent programming easier, but I think you're doing many of your users a disservice by not exposing those issues up front.

"For example: 'You need to decouple static binding between modules in order to get editor performance...' Will you have a few things slow down at 500 files? Sure. But is it significant? Not when compared to not getting the job done at all."

Couple things here: First, that isn't quite what I was trying to say. In my experience the editor slowdowns are related to excessive and poorly managedstatic binding. I speculate it has something to do with circular dependencies, but as far as I know the real cause isn't known. (Else NI would have fixed it, yes?) If a user follows the hierarchical messaging pattern recommended, those slowdowns are delayed for a while. If the user does not follow hierarchical messaging or they are working on a large project they may find themselves stuck in IDE molasses and have to make huge changes to their actor implementations to get out of it.

Second, 500 files seems like a smallish project for an AF app. If that's when IDE slowdowns start to occur I would hope that information is prominently acknowledged in a FAQ or some other document people are likely to encounter when they first start investigating if they should use the AF.

Third, your comment, "[Is the IDE slowdown] significant? Not when compared to not getting the job done at all" is a false dilemma. It isn't a question of choosing between suffering from IDE slowdown or not finishing the job. It's matter of figuring out how to finish the job without suffering a slowdown at all. For the drowning developer, maybe the slowdown doesn't matter. Evidence suggests it matters a lot to everyone else.

"I'm trying to offer them the AF as a way to handle that sudden jump in complexity in a way that gets them working without making the documentation a six week course in architecture."

I believe keeping the AF documentation focused on how to use the framework (as opposed to general AOD principles) is in fact the correct thing to do. (And that's why I said "NI" in my first post, because I don't think it should necessarily be your responsibility.) But I also believe (because the Labview community is small and learning material is relatively hard to come by) it is in NI's best interest to put some effort into teaching AOD principles, or at the very least encourage users to learn the principles.

"Their presentation will NOT be an introductory presentation for new AF users but will instead be an advanced presentation for architects."

I am intending my content to target those who aren't quite sure what it *means* for something to be an actor, and don't understand how AOD differs from traditional QSM concurrency models. I intend to show some example code illustrating certain concepts, but I am specifically trying to avoid too much code because people get caught up in the implementation details. Once they understand the concepts they can work out the details for themselves in a way that suits their requirements.

IMO the topic is not suitable for new or inexperienced LV users, because concurrent programming is an advanced topic.  I think the topic is entirely appropriate for experienced LV users who are struggling with concurrency or investigating the AF.

[Daklu]

JackDunaway wrote:
It's just an observation that efforts pushing AF have generated positive effects that probably could not have been achieved by pushing [only] AOD.

Probably true, but it's not an either/or proposition.  I don't see any reason why NI cannot promote both.  What's wrong with NI saying...

"AOD addresses the many problems people have had with concurrency.  The principles of AOD are: a, b, and c.  To help you get started doing AOD, we have provided you with the Actor Framework.  Some of the issue you may encounter using the Actor Framework are:  x, y, and z."

[Daklu]

AristosQueue wrote:
I want to see it become a systemically used right answer because that will open up a lot more utility for it (imagine how much work would be saved if every piece of hardware came with an already rolled actor framework API ready for communicating with it).

The same could be said of any widely used actor-oriented implementation.  The question you're answering is, "Would wrapping hardware APIs in an AF layer save time compared to what we have now?"  That's the wrong question.  The question you should be answering is, "Is the AF the correct framework to build an entire actor-oriented ecosystem around?"  You seem to believe it is, presumably because you place a high value on the inherent safety it provides.  I claim it isn't, because the safety you value so highly also prevents advanced developers from extending it in ways they need to to address their specific circumstances. 

We can't make intelligent decisions about what we're willing to give up to meet our goals because the framework is locked down and prevents us from making those tradeoffs.  The AF is presented as a complete package and we can either take it or leave it.  How much more useful would it be for advanced developers if it were instead built on top of other layers that were also exposed to us?  The AF could still provide the structure for the middle of the pyramid developers you're targeting (though I believe it would also be somewhat less protective) and the advanced developers could be free to make those reasoned tradeoffs.

Contrary to how my comments may come across, I don't think the AF is bad.  That's not it at all.  It's that I think the AF could be so much more.  An NI supported actor API is the best chance to unite the LV community and create a flourishing actor-oriented ecosystem, but it has to support use cases and requirements frequently encountered by both mid-level and high-level users.  Designing the AF in a way that effectively shuts out a significant fraction of high level developers feels like a missed opportunity.

I've said all that before and your stance has been that the community benefits more by building protection into the AF.  That may very well be true since the high level developers are, by definition, a minority.  However, ignoring the concerns of your CLA-level developers heavily reinforces the perception that Labview is not a serious language for advanced developers, even among some of those CLA-level developers.

[AristosQueue (NI)]

Daklu wrote:

Designing the AF in a way that effectively shuts out a significant fraction of high level developers feels like a missed opportunity.

Give me more about this. To me, the AF was designed for high-level users -- myself, for example! I don't know of a use case that the AF rules out. I do know of cases that the AF makes you be explicit about, but not that it prohibits. Does it need a dynamic registration layer? Yes. Still working on that. But that's not going to significantly change the existing AF architecture.

Daklu wrote:

We can't make intelligent decisions about what we're willing to give up to meet our goals because the framework is locked down and prevents us from making those tradeoffs.  The AF is presented as a complete package and we can either take it or leave it. 

It's presented as a complete package because there's nothing to separate out. The AF defines an actor class, a message class and the queue class. If you replace any one of those three with your own definition, the entire system works differently, sufficiently so that it is not interoperable with any other actor system. Replacing any one of those three means releasing a whole new framework in its entirety, one that plays by new rules and makes new guarantees. You can either take the AF or leave it because there's no other technical option. Trying to take just a subset of it makes it a different framework.

[AristosQueue (NI)]

Daklu wrote:

"Is the AF the correct framework to build an entire actor-oriented ecosystem around?"

No.

But if you change the question to "Is the AF *one* correct framework to build an entire actor-oriented ecosystem around?" then my answer changes to "Yes." I wouldn't have built it the way it is and I certainly wouldn't have released it as part of LabVIEW if I didn't believe the answer was, "Yes," both from theoretical judgement and from empiric use by users in various projects.

[Daklu]

AristosQueue wrote:

Give me more about this. To me, the AF was designed for high-level users -- myself, for example!

I don't deny that the AF can be used successfully by high-level users, and I know you went to great lengths during the long incubation period to prove the design works in the real world.  The current incarnation of the AF, imo, absolutely excels at helping mid-level developers, unexperienced in writing concurrent code, create small to medium sized concurrent applications safely, provided:

1. They understand dynamic dispatching, and
2. They don't have complex deployment or dependency requirements, and
3. They're okay using the framework as a recipe instead of really understanding what it's about.

While I believe you wrote the AF intending for high-level users to use it and they can absolutely use it successfully, I also believe the restrictions you've imposed and design decisions you've made to make the AF more accessible to the mid-level user are the same things that drive many high-level users away from it.

IMO, one of the biggest differences between a mid-level developer and a high-level developer is a mid-level developer focuses on delivering the functional requirements; a high-level developer focuses on delivering all the requirements, even those the customer hasn't explicitly specified as requirements.

I've frequently mentioned my development style is very organic and how the AF doesn't lend itself well to my dev process.  In large part I've adopted this style because that's what my customers need.  They don't know exactly what they want the app to do or how they want it to do it, and they won't know until they get some time using it.  So one of the requirements for my projects is "very easy to refactor."  Can the AF satisfy that requirement?  No, not to my satisfaction.

When I finish a project I turn the source code over to the customer.  One almost universal non-functional requirement is for them to be able to maintain the code I've written.  If I'm choosing between an AF design and the "just enough" actor designs I advocate that contain familiar implementation patterns, which one do you think I'll choose?  I can walk them through the "just enough" implementation and explain the key principles of AOD in one fairly short session.  How much more time will it take me to do the same with an AF app?  Much, much longer, because I have to spend so much time explaining the the AF.  It's too much new information for them to absorb in a short time period and they will forget how it works before their next editing session.

At the end of the day I need to deliver a solution that meets all of my customer requirements.  The AF is an obstacle to get past, not a clean path to the solution.

AristosQueue wrote:

I don't know of a use case that the AF rules out. I do know of cases that the AF makes you be explicit about, but not that it prohibits.

Yes, the AF allows one to implement any arbitrary functionality they might want.  I'm not disputing that.  I also don't ascribe it much value because one can also implement arbitrary functionality writing x86 assembly code.  People don't for many reasons, one of them being that it's too hard.  They can achieve their business goals more quickly and more easily using other languages.  The same is true of the AF.  It's too hard to meet my and my customer's business goals using the AF and there are far easier ways I can meet them.

AristosQueue wrote:

Replacing any one of those three means releasing a whole new framework in its entirety, one that plays by new rules and makes new guarantees.

Right, I understand that you cannot expose any of those things without breaking the guarantees the AF provides.  It's those guarantees that are the problem.  For the mid-level developer facing tight deadlines and a lack of experience those guarantees can help him meet his requirements.  For some subset of advanced Labview developers we neither need nor want the guarantees provided by the AF.  We have the experience and expertise to manage the risks those guarantees are designed to eliminate in other ways.  We could use the AF if we wanted to, and we might like to tap into some of the potential functionality being offered, but the guarantees provide no value to us and the cost of dealing with and working around those guarantees is too high. 

I think our disagreement over the design boils down to the value of those guarantees.  You clearly place a very high value on them.  Undoubtedly some developers do as well.  If the guarantees are valued then it's worthwhile to deal with the pain those guarantees inflict on you.  We clearly value the guarantees far less than you do.  For us, the cost/benefit tradeoff doesn't come anywhere close to favoring the AF. 

AristosQueue wrote:

But if you change the question to "Is the AF *one* correct framework to build an entire actor-oriented ecosystem around?" then my answer changes to "Yes."

What if we changed the question to "Is the AF the *best* framework to build an entire actor-oriented ecosystem around?"  Of course "best" is open to interpretation.  I guess I'd take it to mean "does the design provide the greatest amount of utility to the greatest number of users?"  Do you still believe the answer is yes?

I've maintained the AF is a good framework, and I think it is for those projects that fit the requirements it imposes.  However, I also think you perceive the practical set of projects for which the AF is a good fit to be much larger than I perceive it to be.  Now that you've expressed the vision of an actor-oriented ecosystem (and I do like the idea) I am feeling a sense of urgency in trying to get you to understand why I think the current AF incarnation isn't the general purpose and widely applicable concurrency solution you seem to think it is.  Building an ecosystem based on the AF only addresses the needs of those projects for which the AF is already a good fit.  It does nothing for the rest of the projects. 

I have the highest amount of respect for you, Allen, and Eli, and all the work you have done on the AF, but I have also been very outspoken about my dislike of the limitations it places on users.  From time to time I get personal messages from people who agree with what I'm saying.**  For whatever reason they choose not to actively participate in the discussion and instead express appreciation that I continue to advocate a position they agree with.  I may be the noisiest and most annoying, but I know I'm not alone, and I perceive the size of the group of people who are discovering the AF doesn't work well for them is increasing.

(**Full disclosure: I honestly do not remember how many times I've been contacted.  Almost certainly less than 10 in the last year.)

[AristosQueue (NI)]

Daklu wrote:

>  What if we changed the question to "Is the AF the *best* framework to build an entire actor-oriented

> ecosystem around?"  Of course "best" is open to interpretation.  I guess I'd take it to mean "does the

> design provide the greatest amount of utility to the greatest number of users?"  Do you still believe the

> answer is yes?

Yes. Of those that exist currently, yes. Dmitri has some proposals that are intriguing, but I know of no way to go after his suggestions without moving the AF out of the usability range of most programmers. I have fielded lots of variations on the forums, public and private, and we have changed AF around in significant ways as a result of those proposals. Yes, I believe this is the single best balance of power and learnability for highly parallel multimodule intercommunications systems existing in LabVIEW today.

Now, having said that -- I don't necessarily care if people say the AF is not for them. I want them to try it because if they're buidling one of these systems I do believe it is the right choice, but if they ultimately reject it, that's fine. They don't have to use it. That doesn't mean we shouldn't try to make things work as well as possible for those who do use it.

> but I have also been very outspoken about my dislike of the limitations it places on users.

I am aware of your dislike. But the moon is a harsh mistress, and you are somewhat dangerous in that regard.

We aren't building acquire-analyze-present loops when we are reaching for tools like the AF or JAMA or LapDog. We are building highly parallel mult-module intercommunicating systems. No one -- not you, not me, not anyone on the forums, not anyone I've read in literature -- has any proposals for making such systems generally testable, replicatable or debuggable. There is a class of bugs that are nigh impossible to ferret out once they are introduced into the system. That leaves one and only one mechanism for building these systems to have any level of trust: proof by construction. You establish a set of rules that allow you to prove aspects of your system because the only real testing you're going to have of these systems is code inspection. There are simply too many timing options to ever test all the combinations exhaustively.

Proof by construction requires rules. It requires conformity to as much homogeneity as possible even if that homogeneity is a bad fit for a particular module. The health of the system as a whole outweighs the health of any one module up until the point that that module absolutely cannot do its job. And the AF is not that tight of a box -- it allows flexability when a particular communication requires it.

You say that there are advanced developers who don't need the guarantees that the AF provides. I disbelieve you. I've seen too much code written by devs who claim that ability, and that code inevitably has some subset of the four major errors that the AF guards against: the mutual-request deadlock, the missed stop signal, the infinite echo chamber, and the two-communications-channels-deliver-out-of-sequence-messages-leading-to-undefined-state bug (yeah, still looking for a shorthand for that last one). And the more complex the code you're building, the less I believe you. I do believe that in some small projects, you can do it. Great. Don't use AF for those. But I don't believe you as it scales up. And I do believe there are special cases that need special handling. Then make the exceptions exceptional. The AF allows for this.

In short, the AF is ther for those projects that need those guarantees, whether written by novices or architects. And the bigger the scale, the more you, me, all developers need those guarantees.

Now on one hand you speak of advanced developers; on the other you speak of handing off your code to less-experienced clients to maintain -- exactly the folks who you agree clearly benefit from the guarantees. The AF places no limits on the advanced developer -- if a particular module has special needs, then build an actor shell around it and let it be as special as it wants within its scope. But it doesn't get to poison the rest of the system. That means when you hand off the system to less experienced users, the special part is contained. They benefit from the clear bright lines of where things go, which part is responsible for which activity. The OO nature is a blessing not a bane -- numerous universities have now demonstrated OO as the easier programming paradigm to teach, and that's part of the  reason that it is the standard for development in the world outside of LabVIEW.

> I perceive the size of the group of people who are discovering

> the AF doesn't work well for them is increasing.

Well, the community on the forums has gotten everyone unstuck who has been willing to post their problem. Does the AF have parts I'd like to see improved? Yes. Needs more rescripting tools, more debug tools, more example programs. The AF may very well not be for some developers in its current state. That doesn't mean it isn't the right path for us to be putting people on. And if users are speaking to you and not speaking up on the forums -- and you aren't posting their code for the rest of the community to look at -- then there's nothing I can do to improve AF or to change how we teach it to accomodate them.

But my perception is different... the raw number of people discovering AF doesn't work for them is rising, but as a percentage of the population working on these problems, it's remaining about the same. And a good portion of those don't use LVOOP in the first place. For projects of AF scale, not learning LVOOP is foolish. Sure, you can build software of scale without it -- it's been done for years. But there's a list a mile long of reasons why 90% of the world's software these days is OO of some language, and at the top of that list is the number of projects that fail without the organization of code that OO brings to the table. After almost 7 years and the litany of success stories from LabVIEW developers, I have little sympathy.

In short, don't tell me there are situations that can't be solved -- and solved well -- with the AF without posting those situations so we can look at them.

-- AQ

[Daklu]

It seems I've touched a nerve, and I apologize for that.  As near as I can tell I've not only failed to accurately communicate the reasons the AF doesn't work for me, and failed in trying to present a convincing case for a design that everyone can benefit from, but I've managed to tick you off in the process.  I'm sorry, that has not been my intent.