Can I stack up multiple producers and consumers?

[OS-Prof]

Can I connect a bunch of producers and consumers in series, where each one is a producer for the next and a consumer for the previous one? They would communicate via separate queues, because the data types are different from one to the next.

My application collects a bunch of signals from a Multifunction DAQ, and these need to be processed pipeline fashion while subsequent samples are being collected. I would like each of the pipeline stages to be a separate VI, in order to preserve stylistic sanity.

What are the limitations of a series of producers and consumers, in terms of numbers that can be connected in series, performance, etc.?

Thanks,

Hugh L.

[MichaelAivaliotis]

There is no physical limitation. I'm not sure I understand your implementation. Producer/Consumer usually means two independent processes that pass data via one of many possible communication mechanisms. Are you implying that the consumer will also be a producer for yet another consumer? I don't see a problem with this. I guess the real question is, why the need for this complexity? Can't you just string a bunch of VI's together? The wire provides your sequencing and the VI provides encapsulation.

Message Edited by Michael Aivaliotis on 05-16-2007 09:22 PM

[OS-Prof]

The whole application is a real-time system that processes about a thousand signals and reduces them to a handful of numbers indicating physical position of one or more objects. The physical position needs to be updated at the sampling frequency.

At each sampling interval, an array of data is read from the DAQ. This array contains one sample from each separate signal. (Our own circuitry scans the signals to get them into the DAQ at the appropriate times.) The array needs to be pushed through a pipeline of processing stages for filtering, averaging, and other processing, all the while the DAQ is accumulating another array for the next sampling interval. We will apply as many processors as needed to the stages of the processing pipeline in order to get the required performance.

Most processing stages need to retain state from a handful of previous samples in order to accomplish the required filtering or processing of a current sample. Since samples are organized into arrays, this means retaining previous arrays from previous sample intervals.

In my first attempt to structure this application, I found myself embedding the DAQ Read and all of the processing stages in a humungous loop, with lots and lots of shift registers to keep the necessary retained state from one iteration to the next. This got very unwieldy, and I had only got about 1/3 of the processing stages working before realizing that the architecture needed to be rethought.

At that point, I re-educated myself about queues, and now I am trying to design an appropriate architecture to solve this problem. I am leaning toward partitioning the problem at boundaries where concurrent processing makes sense. Each group of related pipeline stages would receive its input from a previous group via a queue and would communicate to the next group via a different queue.

Hence my question:-- what are the issues and limitations surrounding multiple producers and consumers?

Regards,

Hugh L.

[unclebump]

Take a look at the point by point vi's in the analyze section. Maybe even the DSC Tag engine to store the data rather than shift registers. I'm not sure how fast your are reading the DAQ data. Could you do a block diagram or something for your concept. It sounds like some very detailed planning could save you a ton of work later in the wiring.

[OS-Prof]

Can I use the point-by-point VIs on arrays of input signals?

The last three times I read the point-by-point documentation, I came away with the impression that they can handle a stream of scalar values. In my case, I have an array of streams of scalars. How do I process them? Is there an example? I could not find one.

Hugh L.

[unclebump]

Wire your array to the left side of a for loop. Put the point by point vi inside the for loop. Then use the array from the right side of the for loop.

[OS-Prof]

I think that we must be discussing at cross purposes. I could not find any evidence or suggestion in the Help documentation or Knowledge Base that a point-by-point VI can be configured to process a number of signals in parallel, retaining the state of each signal separately.

(Yes, I understand that they are re-entrant, so that multiple point-by-point VIs can be used in the same block diagram. However, my number is of the order of a thousand, and it varies from one run to the next, depending upon the input configuration.)

Nevertheless, I tried coding up a simple example in the attached VI. In this example, I generate an array of N sine waves, each with a different frequency and each with 1000 samples. I then transformed the array, so that I can present the zeroth sample of each wave in parallel, followed by the first, followed by the second, etc. This mimics the DAQ acquiring data for my thousand signals in parallel, one sample per signal at a time.

I then tried to put these samples through a point-by-point integration function using the loop as proposed. It did not work. The point-by-point function could not keep track of a bunch of different states of different signals in parallel, which is what it needs to do.

Regards,

Hugh L.

[Marc_A]

Write your own! Copy the point by point VI you want to use to your own directory and give it a different name. Change the input to an array. Then add a for loop inside the outermost, single iteration while loop so that the calculation is done on each element of the array and this new array is stored in the shift register that used to contain a scalar. This will retain the previous state of all your different signals inside the point by point VI as an array.

That was kind of confusing to write, but I hope you get what I mean.