Feature Request: add Get Queue Status method

[carlod80]

I think it would be useful to have the Get Queue Status VI available also for the DQMH message queue object, like this:

Is there any specific design decision for which this was not implemented yet?

[psmorris]

I'm going to guess the reason it wasnt implemented is that there's explicit need for it in the basic DQMH. That said, I have implemented something similar (with some extension beyond it) in a child class of the delacor message queue (or whatever the actual name of the class is!

I have a monitoring version of the class which maintains a "queue status", or more usefully, the activity status of the message handling loop (i.e. busy or idle). I extended the basic delacor class with a "queue status" method which calls the get queue status VI to get the number of elements left in the queue after a message has been handled. (I also override the dequeue message.vi). I'm hoping to write up what I did and either post it here or somewhere for general viewing.

So, although its not there directly, it's relatively straightforward to add the functionality if you need it, and I think it should be possible to update the DQMH template or use your own template to use a child class implementation by default (Fab - help me out here!)

Paul 

[FabiolaDelaCueva]

---

@psmorris wrote:

So, although its not there directly, it's relatively straightforward to add the functionality if you need it, and I think it should be possible to update the DQMH template or use your own template to use a child class implementation by default (Fab - help me out here!)

Paul 

---

Paul,

Thanks for sharing your solution with us. I am looking forward to that post (hint, hint) 😉

Information on how to create a custom template can be found here: 

http://delacor.com/documentation/dqmh-html/DQMHDocumentation.html?AddingaNewDQMHModulefromaCustomT.h...

For DQMH 4.0 (currently in Beta), the scripting tools do work if you have your own child of the DQMH queue class. The current release version has a bug where you would have to temporarily go back to having the parent queue on the wire when creating new events. 

Fab

[carlod80]

Sorry for coming back in on this after a long time...

The idea of creating a child class of the Delacor message queue at the moment seems messy to me, especially since, as Fab said, the scripting tools do not work with such a custom class in the present DQMH version. And I cannot see any workaround since I cannot extract the original LabVIEW queue from the Message queue object since it's private to the class.

I would actually find it very useful to have such a functionality in the basic DQMH, and I think it would also be of general interest. For instance, I have a DQMH module that takes care of storing data from a high data-rate source (another DQMH module), and I would like to know if the queue is overloaded and take corresponding action, e.g. temporarily cache the data and write a larger data chunk later, rather than attempting to write each chunk to disk immediately. Am I thinking this wrong? I'm very open to comments/suggestions.

[psmorris]

I had a simple (if somewhat ugly) workaround for the scripting tools not working with child classes of the message queue - I stuck a diagram disable structure around the initialise queue VI and could then either enable the original class (meaning the class wire going into the event structure was of the type that the scripting could handle) or enable my override class... 

I just had to remember to switch it to the parent when adding events and back to the child the rest of the time... 

We have to remember that the DQMH is a framework toolkit that, whilst it works well "out of the box", is really designed to be modified to fit individual needs by the end user. If you find yourself wanting the same feature that you've added in more than one project, you could look into making a new project template as a starting point... 

Paul

[carlod80]

Thanks for your prompt feedback Paul, that's a good trick!

I still think that DQMH, being a framework based on queues, should include out-of-the-box some kind of basic diagnostic tool on the status of the queue. Just my two cents... I'm no super-duper-LabVIEW hero 🙂

Fab, what are your thoughts on my specific request?

[FabiolaDelaCueva]

---

@carlod80 wrote:

... especially since, as Fab said, the scripting tools do not work with such a custom class in the present DQMH version...

---

you could download the DQMH 4.0 Beta version 😉

---

@carlod80 wrote:

I would actually find it very useful to have such a functionality in the basic DQMH, and I think it would also be of general interest. For instance, I have a DQMH module that takes care of storing data from a high data-rate source (another DQMH module), and I would like to know if the queue is overloaded and take corresponding action, e.g. temporarily cache the data and write a larger data chunk later, rather than attempting to write each chunk to disk immediately. Am I thinking this wrong? I'm very open to comments/suggestions.

---

I might not be understanding the entire problem. What I would do is create a "Log to File DQMH" module. The "Acquisition DQMH" module, would send a request to log data as frequently as it wants. 

The "Acquisition DQMH" responsibility is to acquire data.

The "Log to File DQMH" responsibility is to log to file. 

The "Log to File DQMH" module then decides how often it wants to write to disk and how many elements to buffer in place before writing to disk. One more thing, since this module is responsible of making sure all data is logged, its Exit message would not be a priority message because it needs to write every remaining queued "log data" message. 

On exit, it would write whatever is left in its buffer.

If any of these Requests/Broadcasts are too "chatty", you can always create a helper loop along the lines of what Peter Horn described in his Message Pumps post.

Does this make sense? 

Regards,

Fab

[joerg.hampel]

---

@FabiolaDelaCueva wrote:
If any of these Requests/Broadcasts are too "chatty", you can always create a helper loop along the lines of what Peter Horn described in his Message Pumps post.

+1 for the helper loops. If I find the time I'll write down a little something about it, too. We only recently discussed pros and cons (or rather when's and when-not's) with a customer...

[carlod80]

---

---

The "Acquisition DQMH" responsibility is to acquire data.

The "Log to File DQMH" responsibility is to log to file. 

Yes, this is exactly the architecture I am using now, using Broadcasts (sorry my explanation was too poor). On this point: are there any performance considerations to account for when chosing between Requests vs. Broadcasts?

---

 The "Log to File DQMH" module then decides how often it wants to write to disk and how many elements to buffer in place before writing to disk.

---

This is precisely my point: how can the "Log to File DQMH" decide how many elements to buffer? Note that my data rate from the Acquisition Module can largely change (and become pretty high - up to 50kHz-triggered arrays with 100 DBL points each), therefore I (could) need to dynamically adjust my buffer size. The only way I can imagine is to inspect the queue status and see if it is growing indefinitely (i.e. the data rate is above the storage rate) and consequently increase the buffer size.

As far as I understand, a helper loop would have the same issue, i.e. if data rate exceeds the storage rate then I'll quickly run out of memory without "adaptive buffering".

Note that I'm using HDF5 for storage so I have some overhead for on-the-fly compression of the chunks and calls to the HDF libraries.

By the way: thanks Fab for the neat tip on the "Exit" message, I missed that detail!

---

[FabiolaDelaCueva]

---

@carlod80 wrote:

---

Yes, this is exactly the architecture I am using now, using Broadcasts (sorry my explanation was too poor). On this point: are there any performance considerations to account for when chosing between Requests vs. Broadcasts?

---

---

There are no performance considerations, both Broadcasts and Requests are events. The only performance consideration for both is if you have a very chatty module, then you might need different helper loops with event structures to separate what types of events are handled in which loop. I will post a separate discussion on how to decide if an event is supposed to be a Request or a Broadcast.

---

@carlod80 wrote:

---

This is precisely my point: how can the "Log to File DQMH" decide how many elements to buffer? Note that my data rate from the Acquisition Module can largely change (and become pretty high - up to 50kHz-triggered arrays with 100 DBL points each), therefore I (could) need to dynamically adjust my buffer size. The only way I can imagine is to inspect the queue status and see if it is growing indefinitely (i.e. the data rate is above the storage rate) and consequently increase the buffer size. 

The helper loop would have an event structure registered for the following requests:

* Stop Module

* Log Data

There is no timing in the loop, the loop runs as fast as it cans or it sleeps if there is no data to log.

The Stop Module event case is there to ensure the helper loop can be stopped and also to write any leftover data in the buffer.

In the Log Data, the code checks if the buffer is full, if it is full, it logs to file, if it is not full it continues to fill the buffer up.

I would just pick the worst case scenario as to how big you want that buffer to be and assume that you are receiving data as fast as you can. 

Note that the data is logged there, directly in the helper loop, no need to enqueue back into the MHL.

---

@carlod80 wrote:

---

By the way: thanks Fab for the neat tip on the "Exit" message, I missed that detail!

---

---

I missed that detail too! Once, but that was enough to become a rule in my book.