06-13-2016 12:09 PM
Thanks for a great tool.
I am however faced with the challenge in its application - I want both the multi unit feature of DQMH but each of my modules will be working similarly to the continous measurement and logging (CML) project examples. My PC hosted system has multiple independent hardware acquisition units (ADC, DAC, GPS, Cameras,) which I want to be able to orchestrate from a central controller. They all send their time tagged acquistion data to the same TDMS file (Praise the TDMS file concept for that neat feature!)
The UI messages are used in the CML frameworks acquisition and logging vi's, but CML uses another queue type relative to DQMH so it is not so easy to merge the examples.
I tend to think it is easiest to add CML features to DQMH, than the other way round. But to make it neat I would then like to use the same sort of queue (LVOOP based), but this part resides in the vi.lib and I dont want to touch that. So can I add a new set of message queues with relative ease?
Or maybe I am down the wrong road with my architecture.
Any suggestions are welcome - before I start digging myself into a hole:-)
06-14-2016 09:09 AM
Hi Heel,
In one of the presentations I will be giving at NI Week this year (The Decisions behind the DQMH design) I will be going over some of these details. But I understand you have the problem now and might not want to wait unitl NI Week
The main difference between the NI QMH (used in the CML and Logging projects) and the DQMH is that there are two communication paths in the DQMH.
The NI QMH uses only queues and the queue to each process is shared via a cluster of queues among all the processes.
The DQMH makes the queue private to the process, only the process can queue messages to itself. If other processes want to communicate with the DQMH module, they have to do it via events.
So the way to "translate" the CML project is to replace any messages sent among processes with events created via the Tools>>Delacor>>DQMH>>Create New DQMH Event...
Also keep in mind that any continuous operation like acquiring data, is better left for a helper loop on you DQMH module.
Now regarding this comment:
heel wrote:
I tend to think it is easiest to add CML features to DQMH, than the other way round. But to make it neat I would then like to use the same sort of queue (LVOOP based), but this part resides in the vi.lib and I dont want to touch that. So can I add a new set of message queues with relative ease?
You can create your own Queue class and just make it inherit from the Delacor_lib_QMH_Message Queueu.lvlcass. If you are using DQMH 2.1, you can override all the methods you need to make your own queue. But try first what I mention above, I don't think you will need a separate queue.
If you want to post your attempt at making the conversion, we can point out areas for improvement.
Best regards,
Fab
06-14-2016 10:44 AM
Dear Fab,
Unfortunately I am not going to NI week, but thank you a lot for offering your suggestions.
I am going to:
I may use the opportunity to ask for improvements - once I get that far, but there appears to be a lot of tedious work until then. The DQMH builder does however give good promise of a flexible and maintainable architecture.
br. henning
06-14-2016 01:46 PM
heel wrote:
Dear Fab,
Unfortunately I am not going to NI week, but thank you a lot for offering your suggestions.
No worries, we will eventually post the video of the presentations we give.
Let me start by saying that the CML sample project is excelent if all you are going to do is acquire from a single device and do what the example does. If that is all you are doing, then the DQMH approach might be overkill. However, if you want something that will extend with other modules later, then the DQMH is a better way to start.
Let me make some suggestions to your approach:
heel wrote:
I am going to:
- Start from DQMH project and
Yes, you could also start from a blank project and add the modules that I describe below. The advantage of starting form the DQMH project is that you will already have the build specification and you can define at least one of your modules (for example, rename the "My Singleton" as "Acquisition Module")
heel wrote:
2. Add the acquisition, logging and monitor loops to this project
In this case you would not be creating added loops, instead, you would create new DQMH modules:
* An Acquisition DQMH
* A Logging DQMH
* The Display loop could be replaced by a loop in your top level application that has an event structure inside, registered to listen to the Data update from the Acquisition DQMH module.
heel wrote:
3. Use the queue class already defined in DQMH - simply create further instances (Acquisition, and Logging) similar to the CML project example. These will convey messages between the UI loop, Message handling loop, the Acquisition loop and the Logging loops. Crafted along the architecture in CML projet.
You don't need to do this, instead, you will create new events for your different DQMH modules.
For example, you would create a new Request event for the Acquisition module called "Start Acquisition" with a dummy boolean argument called Start. This could be also a Request and Wait for Event that returns as a reply whether the acquisition started successfully. Don't forget to add an error as an argument for your reply, in case you have to send back what error caused the acquisition to not start.
The Acquisition module will need a helper loop to continuously acquire.
I suggest you do a simple bubble diagram showing with arrows the events that you want to send between all modules. You could also create a sequence diagram.
heel wrote:
I may use the opportunity to ask for improvements - once I get that far, but there appears to be a lot of tedious work until then. The DQMH builder does however give good promise of a flexible and maintainable architecture.
br. henning
Please keep us posted. You might have nudged me to start working sooner than later on the demonstrations for the NI Week presentation. If you are a CLA, I gave a presentation in 2013 at the CLA Summit that describes a very early version of the DQMH (it was not even called that at the time) and how to modify the CML. Here is a link to it: https://decibel.ni.com/content/docs/DOC-29048
If you are not a CLA, you won't have access to that link.
[Edit] I made the slides available via the private section on Delacor.com. You will have to join to get to them, I promise we won't spam you
Here is the link: http://delacor.com/?p=660
You will notice that the steps on slide 30 have been simplified a lot with DQMH:
Slide 33 will look very familiar, it is the CML Main.vi transformed to use DQMH looking API calls.
Slide 35 our advise has changed to use helper loops for repetitive tasks or tasks that need to wait. You don't want to ever hold your Message Handling Loop hostage.
Slide 36 is just describing the "Requesst and wait for Reply" DQMH event that you can create with the DQMH productivity tools (Tools>>Delacor>>DQMH>>Create DQMH User Event...)
I hope this helps.
Regards,
Fab
06-15-2016 09:01 AM
Thanks Fab,
I have a concern about the link between acquisition and logging. It is typical that:
Using only an event message (broadcast or point to point) will not meet this requirement I would think, but I may be wrong. In the CML project the data queue serves as a buffer and solves this issue. So - are there multilevel buffering in the event queue? If not what do you suggest to meet the above requirements using DQMH?
thanks for the great advices you provide.
06-15-2016 09:17 AM
heel wrote:
In the CML project the data queue serves as a buffer and solves this issue. So - are there multilevel buffering in the event queue?
The Event Queue provides the same buffering as a queue like used in the CML. I have used User Events for my logging for a few years now with no issues at all.
06-15-2016 11:44 AM
crossrulz wrote:
heel wrote:
In the CML project the data queue serves as a buffer and solves this issue. So - are there multilevel buffering in the event queue?
The Event Queue provides the same buffering as a queue like used in the CML. I have used User Events for my logging for a few years now with no issues at all.
Exactly, at the end of the day the "Register for Events" launches an event engine that "dequeues the events" and handles them one by one in the event structure.
This is one of the reasons I prefer events over queues. If the module uses the queue as a message transport and another module enqueues the message but the module is gone, the enqueing of the messages becomes a memory leak. However if the events are used as the transport mechanism, if a module fires an event and there is no module register to listen to it, no problem, it does not get enqueued anywhere. If there are two modules listening to it, then it will get enqueued in two engines.
06-15-2016 02:43 PM
Cool. I try to write down my understanding.
So if there are N consumers, (and M producers) it correspond to having N queues. It means that the consumers will not necessarily be in sync (=working on the same event) as the queues may hold different number of events at any instant. But one can be sure all issued events will eventually be processed sequentially by all consumers - lest the obvious exceptions of shutdown or starvation. I am very unsure if this holds, as I could not get a small test example with M=2 producers to work.
If one of the consumers shuts down, the rest of the consumers will be unaffected.
If one of M producers shuts down it will not affect the queues as long as M-1>0
Events accumulated while no consumers are registered will be discarded. I.e. no need to flush stale data. (as already written by Fab)
For the consumer to flush the event queue en-route, the consumer has to read and discard each event, or unregister and re-register again.
There is no way to post events to the front of the event queue.
I realize a nice feature of the event queue versus the "normal" queue is that the consumer can be dormant while no events are available but can still be woken in an orderly manner for other actions - shutdown for instance. To solve this with a normal queue you have to
1) use timeout in the dequeue and
2) have a separate means of communication to convey shutdown actions
Did I get it right? Very powerful indeed.
06-15-2016 03:05 PM
heel wrote:
Cool. I try to write down my understanding.
So if there are N consumers, (and M producers) it correspond to having N queues. It means that the consumers will not necessarily be in sync (=working on the same event) as the queues may hold different number of events at any instant. But one can be sure all issued events will eventually be processed sequentially by all consumers - lest the obvious exceptions of shutdown or starvation. I am very unsure if this holds, as I could not get a small test example with M=2 producers to work.
If you post your small test example, we can probably help you modify it so you can see this. But yes, all issued events will eventually be processed sequentially by any event structure registered to listen to that event. Please note that there has to be a one to one relationship between the Register for Events node and the event structure. Do not fork the register for events reference.
heel wrote:
For the consumer to flush the event queue en-route, the consumer has to read and discard each event, or unregister and re-register again.
Another alternative, starting on LabVIEW 2013 (I think), the palette for the user events was expanded and it now includes a flush events VI.
heel wrote:
There is no way to post events to the front of the event queue.
There is, starting on LabVIEW 2013, the expansion of the User Events palette included adding a priority input to the Generate User Event VI. If you explore the Stop Module VI in the DQMH modules, you will see that we send that event as a priority event, meaning it gets post at the front of the event queue.
heel wrote:
I realize a nice feature of the event queue versus the "normal" queue is that the consumer can be dormant while no events are available but can still be woken in an orderly manner for other actions - shutdown for instance. To solve this with a normal queue you have to
1) use timeout in the dequeue and
2) have a separate means of communication to convey shutdown actions
Did I get it right? Very powerful indeed.
Exactly! you can completely say bye bye to polling code
06-15-2016 03:28 PM
Fab,
you have telepathic abilties - exacly, I used a copy of the register-for-event reference to two event loops, but like this it works:
a bit subtle. thanks