Properly Manage and Release Multiple Queues in QMH

[tsr26]

Hi,

I have written an application for controlling two motors on a rehab device I am working on for my PhD. 

It acquires data, processes it, logs it, displays it, and can go through various 'protocols' as our experiments define them. 

I have based the architecture off the LabVIEW sample project 'Continuous Measurement and Logging' - the QMH style.

I am getting behavior where occasionally one loop (most recently loop 7) will not Shutdown properly. I have verified with breakpoints that the message is being enqueued, but it is going back to the 'Process Data' state immediately afterwards.

I have read bits and pieces across the forums about a possible reason - that I enqueue a loop's state within that loop so it stays in a certain state until I say otherwise. I do that because that is what the LabVIEW template does. I assume my problem may be some sort of race condition between the 'misbehaving' loop and the 'Main Brain' Loop.

1) Is there something wrong with the way I am releasing the queues? I could not figure out why the template from LV released all the message queues within their respective consumer loops, but didn't release the data queues until after the UI Message loop finished.

2) If the implementation of enqueuing the loop's state inside itself is causing these race conditions, how should I change it/avoid it? I have read from 'crossrulz' that using the 'timeout' feature works, but I just can't envision what that actually looks like (and if it would create any other necessary structure changes for my loops)

The attached project is in LV2015. I apologize for any curse words in the 'Help Doc' I made...right now only I see it and sometimes I use it to vent 🙂

Thanks for being generous with your time,

Tom

PS - all other criticisms are welcomed as well (non-personal ones of course)

[Gregory]

I don't think the issue is due to where you are releasing the queues.

I would try to avoid a re-enqueueing the same state over and over, because then you have to enqueue your shutdown "at other end" to try to make it run first. It should be fine, unless there is a bug with that function.

How do you know a loop is not exiting? You see the loop counter continue to increment? And the loop that continues running seems to be random?

I would not have the "shutdown" enqueue functions all linked together by error wire. The shutdown command is pretty important, and you want it to run whether there is an error or not.

[tsr26]

Gregory,

I have read that I should avoid re-enqueueing the same state, but I'm not sure how to implement a viable alternative 😕

As for the loop not ending - I'm now questioning myself but here is why I thought it wasn't exiting:

1) When I hit 'Stop Program', the program does not stop

2) The indicator I have off the dequeue for the Processing loop does not say 'shutdown' like every other loop does. The iteration does not increase, but I figured it was because the loop was waiting on a dequeue that wasn't coming...

3) I put a probe off the error wires coming out of the 'misbehaving' loop and I noted that the wire had not executed although all other loops had.

As to not wiring the 'shutdown' cases together - that makes sense! I'll remove that thanks. Now I'm wondering if that was part of the issue...However it is hard to be sure since the program not stopping is sporadic. I've had this issue with the Logging and Processing loops seemingly at random. I have been trying to figure out the cause.

[Gregory]

Sorry after looking at it again, I think I have a good guess as to what's going on.

As soon as loop "4) Data Acquisition Loop" stops, the "Processing Data Q" no longer gets data fed into it.

When loop "7) Data Processing Loop" goes into the "Process Data" case, it will wait indefinitely for an element to go onto the "Processing Data Q".

The way I use QMH's, the command and data are in a cluster coming from the same place. Unfortunately, in your program the command comes from one place and the data comes from somewhere else, and they are not in synch. It is quite possible for your data generator (loop 4) to be shutting down right as your consumer (loop 7) is starting to process data, but it's left hanging.

[Kevin_Price · Accepted solution]

You have a lot of self-enqueuing behavior going on.  Some would say you should NEVER do that.  I'm a bit more lenient with myself and will *sometimes* do it as a shortcut, but I definitely recognize the hard-won wisdom in the NEVER advice.

Attached is a very bare-bones illustration of how to repeat a state without re-enqueueing.  You set a non-infinite Dequeue timeout, and then repeat the desired state whenever the (often expected) timeout happens.  The timeout can be very short, you just want to allow your "main brain" the opportunity to put the "shutdown" msg into the queue.

But all that being said, I think your shutdown problem is due to something else.  I can't take the time to trace all of the logic and sequencing, but here's what I suspect:

In Loop 7, the Data Processing Loop, your repeatedly self-enqueueing "Process Data" case contains an interior Dequeue with an infinite timeout.  It waits for something to arrive from Loop 4, the Data Acquisition Loop, over the "Processing Data Q".

Probably, the Data Acq Loop processes the Shutdown message while the Data Processing Loop gets stuck waiting for data in the Processing Data Q.  Meanwhile, the shutdown message *IS* placed in its own main queue, but the loop never gets a chance to iterate and handle it.

The various self-enqueuing operations, Dequeues with infinite timeouts, and sequencing of shutdown messages can all interact to make this kind of race condition happen.  The first step to "unclog" the shutdown process is for any *interior* Dequeue operations to have a finite timeout that you handle with a case structure.  (When timeout is False, run the normal code, when it's True do nothing but proceed to the next iteration to do your outer Dequeue.)

You mention that you see shutdown issues with Logging and Processing Loops?   Well, those are the ones that contain interior Dequeues with infinite timeouts.  So that lends more credence to my theory.

-Kevin P

[billko]

---

@Kevin_Price wrote:

You have a lot of self-enqueuing behavior going on.  Some would say you should NEVER do that.  I'm a bit more lenient with myself and will *sometimes* do it as a shortcut, but I definitely recognize the hard-won wisdom in the NEVER advice.

Attached is a very bare-bones illustration of how to repeat a state without re-enqueueing.  You set a non-infinite Dequeue timeout, and then repeat the desired state whenever the (often expected) timeout happens.  The timeout can be very short, you just want to allow your "main brain" the opportunity to put the "shutdown" msg into the queue.

But all that being said, I think your shutdown problem is due to something else.  I can't take the time to trace all of the logic and sequencing, but here's what I suspect:

In Loop 7, the Data Processing Loop, your repeatedly self-enqueueing "Process Data" case contains an interior Dequeue with an infinite timeout.  It waits for something to arrive from Loop 4, the Data Acquisition Loop, over the "Processing Data Q".

Probably, the Data Acq Loop processes the Shutdown message while the Data Processing Loop gets stuck waiting for data in the Processing Data Q.  Meanwhile, the shutdown message *IS* placed in its own main queue, but the loop never gets a chance to iterate and handle it.

The various self-enqueuing operations, Dequeues with infinite timeouts, and sequencing of shutdown messages can all interact to make this kind of race condition happen.  The first step to "unclog" the shutdown process is for any *interior* Dequeue operations to have a finite timeout that you handle with a case structure.  (When timeout is False, run the normal code, when it's True do nothing but proceed to the next iteration to do your outer Dequeue.)

You mention that you see shutdown issues with Logging and Processing Loops?   Well, those are the ones that contain interior Dequeues with infinite timeouts.  So that lends more credence to my theory.

-Kevin P

---

I avoid "self-enqueuing" like the plague.  Either have a state machine that drives itself, or a QMH that responds only to the messenger.  To have a hybrid is asking for trouble of the worst kind.

[tsr26]

Kevin,

I am working through your suggestions and theories now - will report back.

Bill,

To be clear, is the snippet from Kevin the state machine, the 'pure' QMH, or the hybrid? I'm having a hard time sorting that out. I realize the category isn't what truly matters - I just use bins in my head to keep different strategies straight.

Gregory, Kevin, and Bill, 

Would you all recommend I start a topic specifically asking about self-enqueueing behavior in QMH loops? I ask because although I've dug into some threads that ultimately say "don't self-enqueue", I haven't found any direct search results for that nor can I find any QMH resources that dig into the ways to command a loop to start a process and continue it on its own until you tell it to stop. That information seems scattered to me. Let me know what you think on that or if I'm just missing some resources.

Thanks,

Tom

[tsr26]

Kevin,

Another question about your snippet, would using the timeout feature limit my loop speed to 1000Hz? I assume the smallest timeout I can set is 1ms.

Tom

[Kevin_Price]

RE: the snippet

    My snippet is trying to show how you can get a QMH to perform the same action repeatedly while remaining responsive to new messages that may arrive and without enqueuing messages to itself.   I hadn't tried to think carefully how to categorize or label it, but I suppose I'd consider it more hybrid than pure.  It has an element of self-determination that I suppose a pure QMH wouldn't have.  Once you send the "Process Data" message just once, it starts behaving like a state machine that keeps repeating the same state.  

    Note that it's incomplete, and some things about it might be misleading.  I probably should have had all cases except "Process Data" wire an explicit -1 value to the timeout shift register to be more clear.

RE: new thread

    If you were to start a thread about QMH's and self-enqueueing, I'd highly recommend that the first post should link to the various relevant but scattered threads you mentioned.   Perhaps you could even add a sentence or two about each to summarize, comment, or whatever else to help distinguish and identify them.

   It sounds like that kind of aggregation could be pretty useful, especially if you can embed several of the keyword phrases that you think would help others find the fuller collection of info easier.

RE: use of timeout and loop speed limit.

   Well, timeout behavior would be limited to 1000 Hz.  But if some external process were feeding messages into the queue faster than 1000 Hz, the loop would run faster than 1000 Hz to keep up.

   But honestly, it's kind of a bad question to be asking.  If you were thinking of doing something in a way that required that knowledge, don't do it.  Think again.  You almost definitely shouldn't want or plan to iterate a timeout case at such rates.

-Kevin P

[tsr26]

RE: Snippet

     Okay that makes a lot of sense - thank you for clarifying.

RE: New Thread

     Perfect - I'll gather it up. Thank you for the advice about effective posting. I'm trying to be as focused and effective on here as I can. This forum has helped me a TON and I'd like to return the favor.

RE: Use of Timeout

    I really don't want to have the timeout feature pace my loop. The problem I'm stuck on is that, since I have 2 dequeues in a loop (one for the state/message of the outer case structure and one for the data being dequeued inside said case structure), I need the state/message dequeue to be as fast as possible. Otherwise the inner dequeue won't execute. That is why I am wondering if using the timeout feature will be a viable option since ideally I wouldn't limit my acq speed to 1000Hz. When I self-enqueue, then both queues get updated at the same rate.

     I believe I also remember that 2 dequeues in one loop isn't a good idea - this probably being the reason why. So I may have to restructure somehow...

Tom