Is there any problem with sending acquired data on the message queue? (QMH)

[Shiv0921]

I feel like there's a rule of thumb here I'm not aware of.

I'm using a QMH using the message queue library.  I need to send acquired data from acquisition to processing. Is it recommended for this to be enqueued on the processor's message queue (message string + data as variant), or a separate 'data only' queue? 

This example could also be for acquisition -> logging. 

I'm sure it doesn't matter a lot but I use QMHs a lot so would just like to clear this up in my mind. 

Data on the message queue:

- Processor can react to receiving 'here's data' as is needed. Means I don't have to repeatedly self-enqueue a 'process data' message that reads the data queue and wont be able to be responsive to a variable acquisition rate.  

- Data forwarding could get caught up with other messaging, but actor framework sends everything on the same queues as messages so surely the actual impact isn't so bad? Not to mention the loop's only role is to process so how many other messages are really going to get in the way.

Data on it's own queue:

- I don't have to convert the data to variant and back. Does this even matter? Is there any drawback to converting data arrays to variant?

I'm leaning towards the message queue but I don't have that much grasp of whether its more about a performance difference. Thoughts appreciated 

[Gregory]

A picture would help a lot here. But, a few words of advice:

1. Acquisition and logging are often done in separate loops, so that the logging does not slow down or alter the timing of the acquisition.

2. Since they are in separate loops, you'd need 2 queues, one for each loop. You cannot dequeue from 1 queue in 2 separate loops.

3. If you start a new project, one of the template options should be a sample project called "Continuous Measurement and Logging". You can start one of those just to see how NI's example handles this use case.

[Shiv0921]

Thanks for replying.. I should have explicitly said acquisition 'loop' and processing 'loop', I'll admit I would have thought the fact that I'm comparing one of the implementations of the actor framework to how I sometimes see QMHs might have escaped me being told to look at that example haha, but yes a solid reminder to always include code or a diagram. 

The question should really be Why does the Continuous measurement and logging example use a separate queue for data?

Here's the Acquisition loop, and then the logging loop (all from the continuous measurement and logging example, which I've attached but obviously we all have)

Acquisition enqueues its read data to a data-only queue (going to Logging).

Logging messages itself to 'log' on its message queue and reads data off of the data queue. 

Why doesn't acquisition just enqueue the data onto the logger's message queue? That way log wouldn't have to message itself to log and can react to data at whatever rate it comes in, state permitting. 

Is the separate data queue because it's not good to convert arrays to variant? (which is what's needed to send data on the message queue) Or is it some worry about clogging up the messaging queue with messages just delivering data? But then, if one queue between actors is good enough for the actor framework, why not QMH?

[Gregory · Accepted solution]

Those are good questions. I've converted large sets of data to variant before and it hasn't been a bottleneck for my applications. 

The best reason that I can think of for the separate data queus is that it makes it a little more obvious that the waveform data is being shared between those loops. This is one of the advertised benefits of channel wires. Since this example pre-dates channel wires, maybe that was the intent.

Personally, I'm not a fan of the extra data queues or the self-enqueuing. I'd use the name of the loops (ex. Data Acquisition and Data Logging) as a strong hint that data is shared between the two. Then comments and good icons on the block diagram to make it crystal clear.

[crossrulz]

In the past, I have just used the same message queue for sending the data.  It makes more sense to me to see only a single queue being maintained by the processing loop.

With that said, the acquisition and logging loops are directly related to each other.  Therefore, in most cases, I would be looking closer at the Producer/Consumer setup and possibly using a Data Stream Channel Wire.  This is also the only place I have found the Channel Wire to actually work well.