DQMH Consortium Toolkits Discussions

cancel
Showing results for 
Search instead for 
Did you mean: 

Merge of DQMH and Continous Measurement and Logging projects

Thanks for a great tool.

I am however faced with the challenge in its application - I want both the multi unit feature of DQMH but each of my modules will be working similarly to the continous measurement and logging (CML) project examples. My PC  hosted system has multiple independent hardware acquisition units (ADC, DAC, GPS, Cameras,) which I want to be able to orchestrate from a central controller. They all send their time tagged acquistion data to the same TDMS file (Praise the TDMS file concept for that neat feature!)

The UI messages are used in the CML frameworks acquisition and logging vi's, but CML uses another queue type relative to DQMH so it is not so easy to merge the examples.

I tend to think it is easiest to add CML features to DQMH, than the other way round. But to make it neat I would then like to use the same sort of queue (LVOOP based), but this part resides in the vi.lib and I dont want to touch that. So can I add a new set of message queues with relative ease?

Or maybe I am down the wrong road with my architecture.

Any suggestions are welcome - before I start digging myself into a hole:-)

0 Kudos
Message 1 of 19
(9,448 Views)

Hi Heel,

In one of the presentations I will be giving at NI Week this year (The Decisions behind the DQMH design) I will be going over some of these details. But I understand you have the problem now and might not want to wait unitl NI Week

The main difference between the NI QMH (used in the CML and Logging projects) and the DQMH is that there are two communication paths in the DQMH.

The NI QMH uses only queues and the queue to each process is shared via a cluster of queues among all the processes.

The DQMH makes the queue private to the process, only the process can queue messages to itself. If other processes want to communicate with the DQMH module, they have to do it via events.

So the way to "translate" the CML project is to replace any messages sent among processes with events created via the Tools>>Delacor>>DQMH>>Create New DQMH Event...

Also keep in mind that any continuous operation like acquiring data, is better left for a helper loop on you DQMH module.

Now regarding this comment:

heel wrote:

I tend to think it is easiest to add CML features to DQMH, than the other way round. But to make it neat I would then like to use the same sort of queue (LVOOP based), but this part resides in the vi.lib and I dont want to touch that. So can I add a new set of message queues with relative ease?

You can create your own Queue class and just make it inherit from the Delacor_lib_QMH_Message Queueu.lvlcass. If you are using DQMH 2.1, you can override all the methods you need to make your own queue. But try first what I mention above, I don't think you will need a separate queue.

If you want to post your attempt at making the conversion, we can point out areas for improvement.

Best regards,

Fab

For an opportunity to learn from experienced developers / entrepeneurs (Steve, Joerg, and Brian amongst them):
Check out DSH Pragmatic Software Development Workshop!

DQMH Lead Architect * DQMH Trusted Advisor * Certified LabVIEW Architect * Certified LabVIEW Embedded Developer * Certified Professional Instructor * LabVIEW Champion * Code Janitor

Have you been nice to future you?
Message 2 of 19
(8,785 Views)

Dear Fab,

Unfortunately I am not going to NI week, but thank you a lot for offering your suggestions.

I am going to:

  1. Start from DQMH project and
  2. Add the acquisition, logging and monitor loops to this project
  3. Use the queue class already defined in DQMH - simply create further instances (Acquisition, and Logging) similar to the CML project example. These will convey messages between the UI loop, Message handling loop, the Acquisition loop and the Logging loops. Crafted along the architecture in CML projet.

I may use the opportunity to ask for improvements - once I get that far, but there appears to be a lot  of tedious work until then. The DQMH builder does however give good promise of a flexible and maintainable architecture.

br. henning

0 Kudos
Message 3 of 19
(8,785 Views)

heel wrote:

Dear Fab,

Unfortunately I am not going to NI week, but thank you a lot for offering your suggestions.

No worries, we will eventually post the video of the presentations we give.

Let me start by saying that the CML sample project is excelent if all you are going to do is acquire from a single device and do what the example does. If that is all you are doing, then the DQMH approach might be overkill. However, if you want something that will extend with other modules later, then the DQMH is a better way to start.

Let me make some suggestions to your approach:

heel wrote:

I am going to:

  1. Start from DQMH project and

Yes, you could also start from a blank project and add the modules that I describe below. The advantage of starting form the DQMH project is that you will already have the build specification and you can define at least one of your modules (for example, rename the "My Singleton" as "Acquisition Module")

heel wrote:

2. Add the acquisition, logging and monitor loops to this project

In this case you would not be creating added loops, instead, you would create new DQMH modules:

* An Acquisition DQMH

* A Logging DQMH

* The Display loop could be replaced by a loop in your top level application that has an event structure inside, registered to listen to the Data update from the Acquisition DQMH module.

heel wrote:

3. Use the queue class already defined in DQMH - simply create further instances (Acquisition, and Logging) similar to the CML project example. These will convey messages between the UI loop, Message handling loop, the Acquisition loop and the Logging loops. Crafted along the architecture in CML projet.

You don't need to do this, instead, you will create new events for your different DQMH modules.

For example, you would create a new Request event for the Acquisition module called "Start Acquisition" with a dummy boolean argument called Start. This could be also a Request and Wait for Event that returns as a reply whether the acquisition started successfully. Don't forget to add an error as an argument for your reply, in case you have to send back what error caused the acquisition to not start.

The Acquisition module will need a helper loop to continuously acquire.

I suggest you do a simple bubble diagram showing with arrows the events that you want to send between all modules. You could also create a sequence diagram.

heel wrote:

I may use the opportunity to ask for improvements - once I get that far, but there appears to be a lot  of tedious work until then. The DQMH builder does however give good promise of a flexible and maintainable architecture.

br. henning

Please keep us posted. You might have nudged me to start working sooner than later on the demonstrations for the NI Week presentation. If you are a CLA, I gave a presentation in 2013 at the CLA Summit that describes a very early version of the DQMH (it was not even called that at the time) and how to modify the CML. Here is a link to it: https://decibel.ni.com/content/docs/DOC-29048

If you are not a CLA, you won't have access to that link.

[Edit] I made the slides available via the private section on Delacor.com. You will have to join to get to them, I promise we won't spam you

Here is the link: http://delacor.com/?p=660

You will notice that the steps on slide 30 have been simplified a lot with DQMH:

  1. Replaced Queue Message Library with class => Done! the queue as you noted before is already wrapped in a class.
  2. Replaced User Event Stop with our version => Done! all DQMH modules come with a Stop Module request event built in.
  3. Removed Enqueue Exit message to other modules => Done! DQMH stop module takes care of enqueing the Exit message.
  4. Replaced Messages to Acquisition Loop with Events => You can do this by creating the Acquisition DQMH Request events needed
  5. Replaced Messages to Logging Loop with Events => You can do this by creating the Logging DQMH Request events needed
  6. Replaced Error messages to Main with Events => Done! All DQMH modules come with their Error Occurred Broadcast Event.
  7. Replaced Data Queue and Data notifier with a Public Data Available User Event and had Main and Logging register to listen to it. => You can do this by creating an Acquisition DQMH Broadcast Event that broadcasts the data received.
  8. Created Launchers for each module, removed them from Main.vi and created Application Launcher. => Done! Each DQMH module comes with their Start Module DQMH VI.
  9. Final touches and editing Build specification and first exe => Done! If you start from the DQMH project template, it already comes with a build specification.

Slide 33 will look very familiar, it is the CML Main.vi transformed to use DQMH looking API calls.

Slide 35 our advise has changed to use helper loops for repetitive tasks or tasks that need to wait. You don't want to ever hold your Message Handling Loop hostage.

Slide 36 is just describing the "Requesst and wait for Reply" DQMH event that you can create with the DQMH productivity tools (Tools>>Delacor>>DQMH>>Create DQMH User Event...)

I hope this helps.

Regards,

Fab

For an opportunity to learn from experienced developers / entrepeneurs (Steve, Joerg, and Brian amongst them):
Check out DSH Pragmatic Software Development Workshop!

DQMH Lead Architect * DQMH Trusted Advisor * Certified LabVIEW Architect * Certified LabVIEW Embedded Developer * Certified Professional Instructor * LabVIEW Champion * Code Janitor

Have you been nice to future you?
Message 4 of 19
(8,789 Views)

Thanks Fab,

I have a concern about the link between acquisition and logging. It is typical that:

  • the acquisition must be running at regular times and as such cannot wait for the consumer - at least not on the short time scale.
  • that the logging must be loss less - no data may be lost.

Using only an event message (broadcast or point to point) will not meet this requirement I would think, but I may be wrong. In the CML project the data queue serves as a buffer and solves this issue. So - are there multilevel buffering in the event queue? If not what do you suggest to meet the above requirements using DQMH?

thanks for the great advices you provide.

0 Kudos
Message 5 of 19
(8,789 Views)

heel wrote:

In the CML project the data queue serves as a buffer and solves this issue. So - are there multilevel buffering in the event queue?

The Event Queue provides the same buffering as a queue like used in the CML.  I have used User Events for my logging for a few years now with no issues at all.


GCentral
There are only two ways to tell somebody thanks: Kudos and Marked Solutions
Unofficial Forum Rules and Guidelines
"Not that we are sufficient in ourselves to claim anything as coming from us, but our sufficiency is from God" - 2 Corinthians 3:5
Message 6 of 19
(8,789 Views)

crossrulz wrote:

heel wrote:

In the CML project the data queue serves as a buffer and solves this issue. So - are there multilevel buffering in the event queue?

The Event Queue provides the same buffering as a queue like used in the CML.  I have used User Events for my logging for a few years now with no issues at all.

Exactly, at the end of the day the "Register for Events" launches an event engine that "dequeues the events" and handles them one by one in the event structure.

This is one of the reasons I prefer events over queues. If the module uses the queue as a message transport and another module enqueues the message but the module is gone, the enqueing of the messages becomes a memory leak. However if the events are used as the transport mechanism, if a module fires an event and there is no module register to listen to it, no problem, it does not get enqueued anywhere. If there are two modules listening to it, then it will get enqueued in two engines.

For an opportunity to learn from experienced developers / entrepeneurs (Steve, Joerg, and Brian amongst them):
Check out DSH Pragmatic Software Development Workshop!

DQMH Lead Architect * DQMH Trusted Advisor * Certified LabVIEW Architect * Certified LabVIEW Embedded Developer * Certified Professional Instructor * LabVIEW Champion * Code Janitor

Have you been nice to future you?
0 Kudos
Message 7 of 19
(8,789 Views)

Cool. I try to write down my understanding.

So if there are N consumers, (and M producers) it correspond to having N queues. It means that the consumers will not necessarily be in sync (=working on the same event) as the queues may hold different number of events at any instant. But one can be sure all issued events will eventually be processed sequentially by all consumers - lest the obvious exceptions of shutdown or starvation. I am very unsure if this holds, as I could not get a small test example with M=2 producers to work.

If one of the consumers shuts down, the rest of the consumers will be unaffected.

If one of M producers shuts down it will not affect the queues as long as M-1>0

Events accumulated while no consumers are registered will be discarded. I.e. no need to flush stale data. (as already written by Fab)

For the consumer to  flush the event queue en-route, the consumer has to read and discard each event, or unregister and re-register again.

There is no way to post events to the front of the event queue.

I realize a nice feature of the event queue versus the "normal" queue is that the consumer can be dormant while no events are available but can still be woken in an orderly manner for other actions - shutdown for instance. To solve this with a normal queue you have to

1) use timeout in the dequeue and

2) have a separate means of communication to convey shutdown actions

Did I get it right? Very powerful indeed.

0 Kudos
Message 8 of 19
(8,789 Views)

heel wrote:

Cool. I try to write down my understanding.

So if there are N consumers, (and M producers) it correspond to having N queues. It means that the consumers will not necessarily be in sync (=working on the same event) as the queues may hold different number of events at any instant. But one can be sure all issued events will eventually be processed sequentially by all consumers - lest the obvious exceptions of shutdown or starvation. I am very unsure if this holds, as I could not get a small test example with M=2 producers to work.

If you post your small test example, we can probably help you modify it so you can see this. But yes, all issued events will eventually be processed sequentially by any event structure registered to listen to that event. Please note that there has to be a one to one relationship between the Register for Events node and the event structure. Do not fork the register for events reference.

heel wrote:

For the consumer to  flush the event queue en-route, the consumer has to read and discard each event, or unregister and re-register again.

Another alternative, starting on LabVIEW 2013 (I think), the palette for the user events was expanded and it now includes a flush events VI.


heel wrote:

There is no way to post events to the front of the event queue.

There is, starting on LabVIEW 2013, the expansion of the User Events palette included adding a priority input to the Generate User Event VI. If you explore the Stop Module VI in the DQMH modules, you will see that we send that event as a priority event, meaning it gets post at the front of the event queue.


heel wrote:

I realize a nice feature of the event queue versus the "normal" queue is that the consumer can be dormant while no events are available but can still be woken in an orderly manner for other actions - shutdown for instance. To solve this with a normal queue you have to

1) use timeout in the dequeue and

2) have a separate means of communication to convey shutdown actions

Did I get it right? Very powerful indeed.

Exactly! you can completely say bye bye to polling code

For an opportunity to learn from experienced developers / entrepeneurs (Steve, Joerg, and Brian amongst them):
Check out DSH Pragmatic Software Development Workshop!

DQMH Lead Architect * DQMH Trusted Advisor * Certified LabVIEW Architect * Certified LabVIEW Embedded Developer * Certified Professional Instructor * LabVIEW Champion * Code Janitor

Have you been nice to future you?
0 Kudos
Message 9 of 19
(8,789 Views)

Fab,

you have telepathic abilties - exacly, I used a copy of the register-for-event reference to two event loops, but like this it works:register event.png

a bit subtle. thanks

Message 10 of 19
(8,789 Views)