An attempt at integrating a synchronous sequencer within an AF code

[WavePacket]

Hi Guys,

I've posted a few times recently about similar topics, but I've come across a use-case where I think I want to embed an AF project with a bit of synchronous code. I'm open to getting pushed off this, but I have given this a fair bit of thought. The use case is a test station where a portion, but not all, of it needs to do a sequence of tests in a specific order.

I'm envisioning a "Hardware Broker" actor which is the node through which all hardware calls get funneled through. Hardware Broker is called by a root actor responsible for handling all the messages to from GUI etc. Hardware Broker normally just forwards messages ala regular AF.

However, Hardware Broker can be sent a "sequence" message where:

(1) In the main message handling loop, *not* in the helper loop, I embed a sequencer using timeout case similar to https://youtu.be/vhOKtshZuUw?t=29
(2) In the timeout case, Hardware Broker synchronously edit with timeout asks root permission to continue to next state (so root can ask for a stop between any states). 
(3) If root gives permission, then Hardware Broker proceeds to next state.
(4) In the next state, Hardware Broker either runs a hardware library call directory, or if a specific piece of hardware has itself been rolled up into an actor, it does a synchronous edit with timeout   wait and reply  to the hardware actor.
(5) When all states have been finished, Hardware Broker fires off the test results to Root via normal message

(6) Hardware Broker completes sequence message, and goes back to normal message forwarding

I realize that I'll lose ability to message all of Hardware Broker's nested actors by this design, but that seems desired. (I want the user to be blocked, or to have ability to change hardware settings based on whether the test is ongoing/not ongoing.)

Anyone care to critique the plan?

[Dhakkan]

Are you trying to do something like this, except with a custom sequencing engine instead of TestStand? If I understand your proposed idea correctly, this engine does the work of your Hardware Broker.

Regarding synchronicity, could you avoid it by introducing a timeout for each relevant step? That way, your actor that is 'waiting' for a message doesn't get locked up; and it can choose to stop waiting when 'important' messages arrive.

Your step 4 would then be the only place where a specific hardware actor makes a synchronous call to its hardware API. The synchronous call itself could be implemented in a helper loop of this hardware actor, if it takes a 'long' time.

[WavePacket]

---

@Dhakkan wrote:

Are you trying to do something like this, except with a custom sequencing engine instead of TestStand? If I understand your proposed idea correctly, this engine does the work of your Hardware Broker.

...

---

Well maybe -- from the the description seems like there is overlap. Unfortunately the description was as far as I could get,  I was presented with an error when I tried to install and VIPM itself didn't return anything directly when I searched for "Measurement" or "Kerry". (I am a bit leery of biting off another framework to digest at the moment, I'm still suffering some indigestion from my recent AF meals.)

---

@Dhakkan wrote:

..

Regarding synchronicity, could you avoid it by introducing a timeout for each relevant step? That way, your actor that is 'waiting' for a message doesn't get locked up; and it can choose to stop waiting when 'important' messages arrive.

Your step 4 would then be the only place where a specific hardware actor makes a synchronous call to its hardware API. The synchronous call itself could be implemented in a helper loop of this hardware actor, if it takes a 'long' time.

---

Thanks for this suggestion. I suppose that for each synchronous message I send off I should attach an appropriate timeout.

Other than those items, any fatal flaws or issues that can be forseen with my proposal?
 

[OneOfTheDans]

You probably have more needs/requirements than described, but on the surface it sounds overcomplicated.

---

@WavePacket wrote:

(3) If root gives permission, then Hardware Broker proceeds to next state.

---

For example, this #3 means that your sequencer isn't Root, and it isn't Hardware Broker... but it's the combination of them. I think having blocking dependencies in both directions (e.g. Root actor 43:stops while Hardware Broker is asking if it can continue) is also a recipe for race conditions.

It might make sense for the Hardware Broker to only perform on-demand hardware access (no sequencing) while the Root Actor owns the sequence. It would send one command (whether from the user, or the sequence profile), then when it gets an OK Msg back from the Broker, index to the next sequence step.

If the sequence is very complicated, maybe Root should control all the hardware directly normally, and when you start a sequence, it passes all the handles into, and launches a Sequence Actor (instead of Hardware Broker). When the sequence is done, you get the handles back in the Last Ack. This approach lets you have different sub-classes for different sequences, if you need that behavior.

[WavePacket]

---

@OneOfTheDans wrote:

You probably have more needs/requirements than described, but on the surface it sounds overcomplicated.

---

@WavePacket wrote:

(3) If root gives permission, then Hardware Broker proceeds to next state.

---

For example, this #3 means that your sequencer isn't Root, and it isn't Hardware Broker... but it's the combination of them. I think having blocking dependencies in both directions (e.g. Root actor 43:stops while Hardware Broker is asking if it can continue) is also a recipe for race conditions.

...

---

That's a good point that I hadn't consider. Humm...

---

@OneOfTheDans wrote:

You probably have more needs/requirements than described, but on the surface it sounds overcomplicated.

...

...

It might make sense for the Hardware Broker to only perform on-demand hardware access (no sequencing) while the Root Actor owns the sequence. It would send one command (whether from the user, or the sequence profile), then when it gets an OK Msg back from the Broker, index to the next sequence step.

If the sequence is very complicated, maybe Root should control all the hardware directly normally, and when you start a sequence, it passes all the handles into, and launches a Sequence Actor (instead of Hardware Broker). When the sequence is done, you get the handles back in the Last Ack. This approach lets you have different sub-classes for different sequences, if you need that behavior.

---

The sequence is very complicated.

For "handles" are you talking about the objects for the hardware? For example, a volt meter class object that reads from a physical voltmeter. I'm not sure what you mean by passing the handles, unbundling the objects from root and injecting them into Sequence Actor? and writing defaults into Root at the same time? (If that's true, Root would have to not accept the messages to control the hardware that it had been controlling just a moment before lest errors occur.)

[OneOfTheDans]

Yes, that's what I meant by handles. Either VISA addresses, refnums, HAL classes, AF enqueuers, or whatever you're using.

Specifically because your sequence is complicated, it seems good to put the sequence alone in its own actor (either self-calling messages, or in a helper loop). No dual purpose hardware-passthrough messages.

Fundamentally, the UI, which talks to its caller Root, is changing behavior when you start the sequence (it becomes unresponsive to the user). I think it's reasonable that Root is the one to implement that change by filtering away the incoming messages, or telling the UI to disable the buttons.

[WavePacket]

---

@OneOfTheDans wrote:

Yes, that's what I meant by handles. Either VISA addresses, refnums, HAL classes, AF enqueuers, or whatever you're using.

Specifically because your sequence is complicated, it seems good to put the sequence alone in its own actor (either self-calling messages, or in a helper loop). No dual purpose hardware-passthrough messages.

Fundamentally, the UI, which talks to its caller Root, is changing behavior when you start the sequence (it becomes unresponsive to the user). I think it's reasonable that Root is the one to implement that change by filtering away the incoming messages, or telling the UI to disable the buttons.

---

Ok, I think I am tracking most of what you wrote. 

Updated idea might be something like this:

1. Root fires up Hardware Sequencer Actor and passes appropriate hardware handles
2. In Hardware Sequencer Actor's helper loop, I implement a case structure like this https://youtu.be/vhOKtshZuUw?t=29
3. In Hardware Sequencer Actor's helper loop, I can register for stop or pause sequence events (and normal stop event  generated in stop core Hardware Sequencer Actor)
4. In the next state, Hardware Sequencer Actor either runs a hardware library call directly, or if a specific piece of hardware has itself been rolled up into an actor, it does a synchronous edit with timeout   wait and reply  to the hardware actor
5. When all states have been finished, Hardware Sequencer Actor fires off the test results to Root via normal message
6. Hardware Sequencer Actor closes itself and the hardware handles are rerouted appropriately in an override of handle last ack core within root (edit to add that I think this would be done by having hardware handles accessible via accessor methods of Hardware Sequencer Actor, and root would know that it was Hardware Sequencer by a to more specific type (link) or by seeing if the enquirer is equal to a list within root (link). 

(As an aside, I'm not sure what you meant by "dual purpose hardware-passthrough messages" but this might be unimportant.)

[OneOfTheDans]

Regarding the dual purpose hardware-passthrough messages: your original design had the Hardware Broker doing double duty (dual purpose). It could either act out a Sequence, or it could forward (passthrough) messages from the GUI intended for direct control.

If you implement your sequence in the Sequence Actor's helper loop, then you'll need all those user events to stop/pause/shutdown.

On the other hand, if you implement the sequence cases as messages of the Sequence Actor, and send-to-self at the end of each function, then your Actor "is" the sequence, rather than "doing" the sequence in the helper loop. You get 1 clean VI per step, which can be nice for version control & debugging, and any messages coming from Root will be naturally interleaved without coding extra events. If you have too many steps for this to be feasible, it could still work with 1 message "run step(int32)" that is just a giant case structure.

But I'm just some guy on the internet; these ideas might not fit your real project.

[drjdpowell]

---

@WavePacket wrote:

(2) In the timeout case, Hardware Broker synchronously edit with timeout asks root permission to continue to next state (so root can ask for a stop between any states). 
(3) If root gives permission, then Hardware Broker proceeds to next state.

---

This is problematic.  It would be better for Root to create a "Sequence Abort Notifier" and provide it along with the Sequence**.  The subVIs inside the Hardware Broker can check the Abort Notifier at multiple points to see if Root has signaled and Abort (can also implement Pause/Resume).  This avoids all your Timeout problems.

**note: a different notifier for each sequence; having only one Abort Notifier would be more problematic and have race conditions.  

[WavePacket]

---

@WavePacket wrote:

---

@OneOfTheDans wrote:

...

---

@WavePacket wrote:

(3) If root gives permission, then Hardware Broker proceeds to next state.

---

For example, this #3 means that your sequencer isn't Root, and it isn't Hardware Broker... but it's the combination of them. I think having blocking dependencies in both directions (e.g. Root actor 43:stops while Hardware Broker is asking if it can continue) is also a recipe for race conditions.

...

---

That's a good point that I hadn't consider. Humm...

...

---

I don't mean to derail the discussion of updated (and still likely better designs), however I was just trying to understand this comment on the older design more fully. I suppose this is getting at a corner of AF that I'm not learned yet on. 

With root in a non-error state -- and it initiates a normal shutdown via 43:stop and asks its nested actors to to shut down -- does root's messaging itself stop right then? I don't know if its yes or no.

• If the answer is yes, then I see the deadlock.
• If the answer is no, then I don't see the deadlock. Root will queue up stop to all nested actors, but still respond to their other messages until they stop. Sequence will continue as root is still responding to "can I continue" from Hardware broker. 

With root in an error state, all bets are off. I also don't know what the default behavior is. Does it send an emergency stop to its nested actors? EDIT -- seems that what happens it that Root's messaging core stops, and then it sends a request for stop to all nested actors. See here: https://youtu.be/PYGo3bEEEPk?t=84. Deadlock certainly possible without timeouts on the synchronous reply messages.