Design Pattern using queues and notifiers and problem with syncing

[KarenM]

Hi

I am rearchitecturing a 4-channel data acquisition system which includes acquiring data, indexing the data between four channels, signal processing,  and data logging.

All processes are placed within a single loop, and I have since separated out the following into a producer and two consumer loop (or master with two slave loops), and use a queue to handle commands, and send the actual data through a notifier list. I am new at this, so I'm not sure how to go about rearhitecturing but am trying to move towards multiple loops. Instead of placing each channel into its own loop, I have used only 2 to save on having to rearchitecture everything.The acquired data is also indexed within the two loops (loop1, loop2):

Main loop: data acquisition (DAQ Assistant) where notifier sends data to consumer loops, queue for general commands

Loop1: reads (indexes 2D array), processes and logs the data for channels 1 and 2

Loop2: reads (indexes 2D array), processes and logs the data for channels 3 and 4

Question 1: Is it ok to use both notifier and queue in the artchiture (best practises)? I would like to use both queues but I find the program may have a lag (even though it doesn't miss any data) and that interrupts the acquisition for the user and I don't want that. I prefer the program to pick up real-time data, and so decided to use notifiers to send data to loops1&2. What are the caveats of this? Since I acquire at effective rate of 1S/s, perhaps this will be sufficient.

As for queueing the commands, this is done and two commands are enqueued in order to activate loops1&2. In using multiple slave/consumer loops, how does one handle multiple channels? the user can select to acquire all channels simulatenously or one at a time. I am able to activate all four channels. In the individual case, since the loops may not execute in order, how do we ensure that the element gets read by the right loop?


I am running into a few problems:
Question 2: data syncing - use 'wait on ms' to sync all loops (main loop, loop1 and loop2)
    - data doesn't seem to be syncing very well - why is this? consumer processing < 1s
    - what am i missing?
Question 3: handling possible data loss - since using notifiers is lossy and using queues is not, I have made it so that commands to so-called 'activate' both loops is done using a queue. I had compared this with notifiers
Question 4: Errors - I made loops 1&2 exit upon error of the notifier error - they throw an error when the program stops so that is the happy case I was testing for. I have read some ni.forums which mention it's sufficient but what are the other types of error to watch out for (notifier error)? Thank you so much for reading!

I have so far programmed everything and it can read all or individual channels but I want to check against all cases, so that's why I'm asking these questions as a check. I know the architecture may not be the best, so any suggestions would be welcome. I'm trying to get the time sync and I don't want any lossy data. I guess using queues for sending both data and commands is ok?

Thanks in advance for your help!

 

[Bob_Schor]

The "idea" of a Producer/Consumer loop is that you have a Producer that "produces" data at (typically) a fixed rate, and you want to do as little processing as possible in the Producer loop so that it is (almost always) "waiting for data", ensuring that you don't miss any data points.  To get the data out of the Producer loop (remember, data produced inside a loop generally stays inside the loop until it exits), you put it in a Queue.  Queues have the ability to almost-instantly "accept" data until they fill up -- by default, Queues have a size of -1, so they'll keep expanding (until you run out of memory) as needed.

The Consumer loop has the (only) Dequeue Element in it.  As long as there are elements in the Queue, the Consumer will take an element off, do whatever is required with it, then return to wait for the next element.  You generally do not need a Wait statement in the Consumer -- if the Queue is empty, the Dequeue will wait patiently for more data to appear.  Note that the Dequeue function has a Timeout input, by default -1 (meaning "Wait Forever").

So suppose that your Producer loop puts data on the Queue at 1000 elements/second.  Let's say the Consumer loop is going to write these to disk -- disk writing is certainly faster than 1000 elements per second, so even if there is a brief interruption in the writing (for example, Windows does something "behind the scenes"), all this means is that the Queue expands a bit, and when the Consumer resumes, it empties the Queue and continues to keep up with the Producer.

Are your 4 channels running synchronously, at the same speed?  If so, consider a single Producer (put a point from each channel, perhaps as an array or a cluster, on the Queue) and a single Consumer (when you get the data, you can deal with the channels as you wish).  If the channels are independent, you might want 4 P/C loops (and 4 Queues).

This is a very useful Design Pattern.  Note that you can get NI to "Explain it All to You" by opening a New VI from the New... (the dots are important!) choice on the File Menu, choosing From Templates, and specifying one of the Producer/Consumer Templates.  Read the Documentation and the Comments.  You do not have to understand (or do) everything that NI shows, but they did put some effort into explanations.

Bob Schor

[KarenM]

Hi Bob,

Thank you very much for the detailed reply. It is much appreciated!

I currently have the one producer and 4 consumers, and actually, rather than logging data, it writes to graph within the consumer and also performs data logging there as well. I originally was thinking to update the graph displays using the producer loop, but getting from the DAQ assistant to the graph is a three step process with considerably intensive calculations at each step. So I thought maybe putting it into the consumers will parallelize the graphical display portion, and allow faster execution. Not the ideal programming practise...although the other option would be to (A) place all 4-channels of graphical display into the producer. I can see how data logging would be very simple using this format (format A) -- however, then the queue would be passing 4 sets of data rather than just 1 set, and thus require use of multiple consumers -- Note: using one single consumer would result in data logging at a rate of 1 S/s, and for four channels the datalogging would be lagging by 4 (and more) loop iterations, unless I'm missing something in the logic or there is another way to do the processing: The queue can be enqueued with multiple (ie. 4) elements per loop, but the consumer processing time would be unpredictable -- how does the architecture rectify this?In terms of synching the consumer, I did read that this was a way to do time synchronization (to put "wait on ms" function on both prod and consumer), but it isn't working for me, and if there is a better practise I'd be happy to use that as well....in short, to ensure the timing of the consumer isn't > than the producer.

(B) Use 1 producer, multiple consumers: Thus, how is it possible to use multiple consumers when each consumer will read the queue at random? Each consumer is supposed to correspond with one channel.

(C) Since channels independent, use 4prod4queues&consumers: I will try this. Thanks!

My program works in the sense of using local variables pass variable information back to the producer to read...to sufficiently save for the timing sychronization and unknown processing time.

[Gregory]

Hi Karen,

The producer/consumer is good at handling short bursts of information. So you can produce a lot of data for a short period of time, and let it slowly dequeue. You do not have to "synchronize" the producer/consumer loops. If you think it might produce more data than can be processed, it is your responsibility to pick a solution. You might want to check the number of elements in the queue and if there are too many, you could briefly pause acquisition. You might want to skip the processing and just save the data to disk, to post process it later.

Applications will often have a "UI Loop" along with the producer/consumer loop. There is a good example of this in the Core 3 course. This way you can send data to the UI Loop and wire it directly to the indicators, instead of having local variables in many places.

For question/scenario (B): The producer loop can send data to 4 different queues, the consumer loops would each dequeue from their own separate queues.

I'd be happy to give some more suggestions if you can post your block diagram.

[Bob_Schor]

Karen,

     I think both Gregory and I are feeling a little frustrated in not being able to answer your questions, largely because we don't entirely understand what you are trying to do (it really would help to see some code ...).  You talk about 4 channels, but don't describe them.  Are they four totally-independent data streams, possibly coming from different DAQ devices, possibly coming in at different rates and timing, and different types of data (i.e. one is boolean, one is an integer, one a float, and the fourth channel is an image from a Web Cam), or are they all "related" (i.e. Channel 0-3 from an A/D converter, sampled at 1KHz)?

     In the first case (4 independent channels, different rates, different data types), it may make sense to have multiple Queues and thus multiple Consumers.  In the "multi-channel Sample", even if the ultimate use of the channels is different, it may make more sense to build the samples into an "array-of-4", and make a Queue for that, sending it to a single Consumer.

     If the Consumer, itself, has to do significant processing of the data, you might want/need to "ship the data" off (via yet another Queue) to a "Consumer/Consumer" (your Consumer becomes the Producer for this second Queue.  For example, I have 16 channels of A/D, acquired at 1KHz, arriving at my PC over the network in "packets" of 50 samples at a time.  As it arrives (in a Producer), it is put on a Queue and sent to Consumer #1.  Consumer #1 does two things with these data -- it passes it to the "Disk Write" routine to write it to the disk, and it puts it on another Queue to be sent to Consumer #2, the Display Consumer.  Here, the data are averaged across the 50 samples and the averaged data from the 16 channels are plotted on a Chart.  Notice that plotting 50-samples averages of 1KHz data means the plot updates at 20 points/second, a pretty useful rate to see things happening in near-real-time and yet have them stay on the screen long enough to actually make sense of what you are seeing (it's very hard to visualize data if you plot it at 1KHz ...).

     However, from your questions, it doesn't sound like what you are doing is this simple.  We'd like to help you, but since we don't understand what you are doing, we can't make particularly useful suggestions.  Give us more information and we'll do our best to help you out.

Bob Schor

[KarenM]

Hi,

Thank you so very much about your replies.

In brief, I am implementing 4 channels of acquisition, where all channels run independently and need ot be acquired simultaneously (in parallel) at a rate of 1S/s each.

I have finally implemented the 4-channel code, using 1 prod and 5 consumer loops:

- prod loop:DAQ asst vi, sends to 2-D array queue

- consum loop: processes UI commands

- consum loop1: processes signal (cannot be removed), graph display, file-write, file-write blinking indicator, event markings

- consum loop2: processes signal (cannot be removed), graph display, file-write, file-write blinking indicator, event markings

- consum loop3: processes signal (cannot be removed), graph display, file-write, file-write blinking indicator, event markings

- consum loop4: processes signal (cannot be removed), graph display, file-write, file-write blinking indicator, event markings

prod loop sends to 4 separate queues and sends 'acquire' commands for every prod loop iteration.This successfully allows acquisition. I have a 5th queue which takes care of UI commands, and a 6th queue to send the 2-D array.

RIght now the 4 consumre loops work fine, except labview doesn't run them in parallel! It takes 4 seconds to go through 1 sample acquisition of all the loops.... why is this?

Sorry for the brevity of my post -- I guess I sorted out a possible methodology thorugh using multiple queues (even though it looks messy).

So now my problem is updated to:

- why is the software not syncing and acquiring at rate of 1S/s, rather it is at 4S/s? data is fed per s from the DAQ assistant...

Thanks so much for your help and perhaps I am complicating things with my explanations. I have tried to make it clear here. Any feedback is appreciated! Thanks again!!!

[Bob_Schor]

Karen,

Suppose I took you to an art museum, blindfolded you, then asked you to critique a painting by an artist whose work you did not know.  That is a poor analogy of what you are asking of us -- you have a reasonably sophisticated piece of code that you would like us to critique, but you have put blindfolds on us because you haven't shown us the code.

I realize that (especially if you've "done it right") there are probably lots of "pieces" (meaning VIs) in multiple folders, hopefully all inside a single top-level folder, even more hopefully inside a LabVIEW Project.  Find the folder with all of the pieces, compress the folder (creating a Zip file), and attach it to your post, saying "Here's the code, look at this VI ...".  It's not that we're not willing to help (and to write long descriptions of how Queues, Notifiers, Producer/Consumer Designs, etc. work), but we are stuck without something to look at and to say "Aha, there's your problem ...".

Bob Schor

[Gregory]

Hi Karen,

Yes, we need to see the actual code to have any idea what's going on. If you have put the loops next to each other, and not nested inside each other, then LabVIEW most certainly is running them in parallel. Whether you have programmed in such a way that causes them to run when you want them to is another question.

It sounds like you have set the loops up to process the data in parallel, but your acquisition loop grabs 1 sample after another serially, resulting in an acquisition time of ~4 seconds.

[drjdpowell]

---

@Bob_Schor wrote:

...or are they all "related" (i.e. Channel 0-3 from an A/D converter, sampled at 1KHz)?

---

That's my guess; the OP is using a single multiplexed DAC card with four DAQ Assisstant Express VIs all configured for individual channels of that one card.  So they can only run one at a time.

[KarenM]

Hi all,

Thank you very much for all your help. I really do appreciate all the advice and detailed explanations which have helped me along the way.

Here is the code: There is initialization on the first sequence frame, then the program enters  a case statement for execution.

The first loop from the top is the main loop

second loop is 'consum loop' which processes UI commands

the following 4 loops are identical, corresponding to the 4 channels.

For the DAQ input, see the second image. Thanks!