Limitations of the Producer Consumer Template

[KurtzC1]

Hello Everyone,

Before I ask my question let me explain my setup. I am using the NI9205 with a load cell, the NI9201 with five pressure transducers, NI9212 with two thermocouples. All of these are connected together in the NI9174 chassis. In order to collect data I am using the Producer Consumer Template to read and write data. In the Producer section I am generating fifteen data points, these data points include the instrument's converted voltage value, the raw voltage value, and time. I am writing all of these to a text file at 500 samples/sec.

My question is, “What is the limit of the Producer Consumer Template?” How many data points do I need to generate per second before the Producer Consumer Template stops working correctly? Surely big companies don’t just use a MASSIVE Producer Consumer Template to collect all of their data. Or do they?

Any thoughts and information would be appreciated!

[crossrulz]

It's not the limitation of the architecture.  It is a limitation of how quickly your consumer can read the data from the queue vs how quickly your producer adds data to the queue.

So if you are sampling at a rate of 500 S/sec, 15 samples will be about 30ms.  So if your data processing and logging can complete within that same 30ms, you will have no issues.  If it takes more time, then you can look into dequeuing more than 1 set of data before logging.

[majoris]

I don't think there is much an issue with the producer consumer architecture in general related to performance. Queues are very time and space efficient. Things unrelated to the acquisition are what is likely to become an issue: CPU can be bottleneck when you have to process the data; disk can be a bottleneck when you have to write data to disk; network can be a bottleneck when you need to send lots of data. At that point, your alternative is to buy more memory to hold all the data until it can be dealt with or buy more machines to do more work in parallel.

[Bob_Schor · Accepted solution]

The point of the Producer/Consumer Design Pattern is to provide a flexible "buffer" (in the form of a Queue) so that data that needs sequential processing, but where some steps might have some "slow" steps that happen occasionally (like the extra processing when first opening a file) can work on the data stream in parallel sharing the processing time so that everyone gets enough CPU cycles so that the entire job gets done.

Typically, one or more elements involve hardware (like A/D converters) that "pause" while doing acquisition (for example, taking 1000 points at 1 KHz takes a second, during which time a parallel task can use the idle CPU cycles).  Usually there is one task (Data Acquisition comes to mind) where you don't want to "miss" any data.  All you need to do is to arrange the subsequent Tasks such that, on average, they can run at least as fast as the "time-critical" task.

You describe a DAQ task that generates 15 data points at a rate of 500 Hz.  So every 2 milliseconds, another 15 points is put on a Queue.  I don't know what you are doing in the Consumer loop (you describe saving the data), but once the data saving is initiated, even if you do fancy formatting, you should be able to output megabytes of data per second, so handling 15 points in 2 milliseconds, or 7500 points a second, should be trivial.  Let's say, hypothetically, that the first call to the Consumer involves opening the file, and since you are using The World's Most Complex Storage Scheme, it takes a second before the file is open.  OK, so you'll have 7500 points stored in the Queue before it can start emptying it into the File.  A second later, the Queue should be empty (since File I/O is Megabytes/sec), so now every 2 milliseconds you empty it again.  No problem!

The only time a problem can arise is if the average speed of the Consumer is less than that of the Producer, so the Queue just keeps growing (as it is being filled by the Producer faster than it is being emptied by the Consumer).  OK, so if you put 7500 points/second in the Queue, and we have an "unlimited" Queue (say we assume you can only hold 750 million points, as your PC only has 4GB of memory), even if your Consumer doesn't run at all, the Queue will be able to hold 100 seconds of data.  This is a pretty ridiculous example because a P/C design where the Consumer doesn't run is Fatally Flawed, but the point is that even a slow Consumer (and a growing Queue) only means that when the Producer stops, the Consumer will (eventually) empty the Queue.  At slow data rates, you should never over-run the Consumer!

Majoris talked about "bottlenecks" -- a bottleneck is only a problem if it is persistent and pervasive.  I've developed numerous routines that have a multiple P/C pathway (A/D, 24 channels at 1KHz, pretty modest) via RT FIFO to ("packaging" in blocks of 50 samples) via Network Streams, TCP/IP to (PC, sent to two Consumers) via Queue to (Consumer 1, average 50 points to create 24 "averages" to add to Chart at 20 points/sec) and to (Consumer 2, stream all 24 channels to Disk File).  No problems at all "keeping up" with this modest data stream.

Bob Schor

Bob Schor

[mcduff]

If you are using the DAQmx API, which I assume you are, you should never be using the Producer-Consumer template for saving data, use the built in logging functions, way, way, way more efficient.

mcduff

[wiebe@CARYA]

---

@KurtzC1 wrote: 

My question is, “What is the limit of the Producer Consumer Template?” 

---

The biggest problem is PCT's encourage a " one-liner" style. So, put all logic in one big VI, and you'll get something that is very hard to debug and impossible to partially re-use. Obviously, you can make proper programs with PCT's, but it's tempting to put all logic in the main. As programs grow, PCT's get more and more difficult to debug. It is for example very difficult to track back where certain elements are enqueued, even if you're lucky nobody put any enqueues in the consumer loop or in sub VI's.

Performance wise, PCT doesn't enforce much limits over the next architecture. Even if there is a bottleneck when enqueueing all elements on the queue, a (circular) buffer would fix that.

---

@KurtzC1 wrote: 

How many data points do I need to generate per second before the Producer Consumer Template stops working correctly?

---

As mentioned, as long as you consume on average faster then you produce, you're ok. That would be highly dependent on the hardware and how you consume. 

---

@KurtzC1 wrote: 

Surely big companies don’t just use a MASSIVE Producer Consumer Template to collect all of their data. Or do they?

---

It's not the size that matters. Big companies make tiny programs. Small companies make big programs. If all the programs the company makes are small (and that is relative) a PCT might be just right...

Arguably, all non-trivial programs have some sort of producer-consumer construct. The only alternative is a one loop program or a one shot sequence. That will only work for tiny solutions, although people keep trying!