Implementing and Understanding the Producer/Consumer Template

[KurtzC1]

Hello Everyone,

I am using an NI6002 to read voltage across two channels. I am using DAQmx to initialize my DAQ and channels. I am storing the read data in memory for approximately 20 minutes through a shift register. Unfortunately, after a couple minutes of storing data in memory my VI noticeably slows down and is unable to write 1000samples/sec.

When I stop the VI and write the data to a text file, LabVIEW errors saying, “The application is not able to keep up with the hardware acquisition. Increasing the butter size, reading, the data more frequently, or specifying a fixed number of samples to read instead of reading all available sample might correct the problem.”

I did some research and apparently a Producer/Consumer Template fixes this problem due to lack of memory. I don’t fully understand why there is a lack of memory. I had Task Manager open when I was running my VI and it did not show the 8GB of available memory being filled. So what exactly is the Producer/Consumer doing? I also am not exactly sure how to implement my current VI into the Producer/Consumer Template, as I have never used one before.

Attached is my VI.

Any advice would be appreciated!

[[dk]]

In your VI, all of your data points are being stored in the shift register and it is just going to keep growing and growing until the loop stops.

What the producer/consumer loop architecture does is offload that shift register from  having to store all of that data. You will have one loop acquiring the data and then transferring it to the other loop via a queue. Then the other loop will be writing that data to file has it is received. This way you are building up a ton of memory in LabVIEW.

What you have now is fine if you aren't going to be collecting data for very long. But 8 minutes of 1000 samples/sec will start to eat up memory. You might even be able to get away with acquiring the data and writing it to file in the same loop (depending on speed) and this would alleviate the memory build up as well.

[[dk]]

If you want to convert your current VI to a producer/consumer you need to:

1- Remove the shift register

2- Drop down a second while loop

3- Share a queue between the two loops

4- Put the 2D array of data that is going to the shift register into an Enqueue Element function

5- Use a Dequeue Element function in the second while loop and write that data to a file

Give this a shot and if you are still having trouble I'll whip an example together when I have time. 

[aputman]

• First of all, your program will run exactly the same with or without the sequence structure.  So get rid of it. 
• The error is actually occurring inside your while loop but since you never handle the error, it only gets displayed after you exit the loop.  The error occurs because you have set the device to acquire samples continuously at a rate of 1000 S/s but you are only reading them from the buffer one sample at a time (NChan 1Samp).  So obviously, the buffer is going to fill up very quickly.  The error isn't related to your system memory (although it will eventually be an issue with an ever-increasing shift register size).
• You only have one channel configured so you don't need to read from N channels. 
  (Edit: you do have two channels configured.  Ignore this point).
• The problem running a program with this type of structure (and by that, I'm referring to the "collect for 20 mins and then write"), what happens if your app crashes or power is cut during the test.  All of that data sitting in the shift register is lost and never gets written to file.  The producer/consumer architecture will address that.

[KurtzC1]

So I tried following your steps as well as reading information online. I think I have the basics of Producer/Consumer Template, but my still VI does not run. It is hard to troubleshoot as I have never done this before.

Other than my VI not working, I had the issue of stopping the second while loop, so I connected both stop buttons to the same button. I don’t know if this is proper procedure. Please let me know.

I am also not sure how to find the maximum and pressure created during the experiment. Previously I could just use “Array Max and Min” but now that all of my data is not an array, I cannot use this method.

Feel free to edit my VI in any way you want and thank you for your help

My new updated VI is attached below.

[Bob_Schor]

I'm attaching a Snippet (in LabVIEW 2016) of a Producer/Consumer example.  In MAX, I created a Simulated 6002 and a Sim_AI Task that has two channels of (simulated) Analog Inputs sampled continuously, 1000 points at a time at 1KHz.  Thus every second a DAQmx Read should give me 1000 points.

In a multi-loop system like a Producer/Consumer, you need a single Stop command (i.e. you need to wire a Stop to one loop).  Which makes sense, to stop the Producer or the Consumer?  I'd argue for the Producer -- you want to "turn off the faucet", not "close the drain".

So what does the Producer do?  Well, there are two things going on -- a DAQmx Read of 1000 points (resulting in a 1D array of Waveforms, as that's what I specified in the DAQmx Read command) and putting this on a Queue (of 1D array of Waveforms, created just before entering the Producer Loop).  We keep doing this until Stop is pushed, when we exit the Producer Loop.  We "clean up" the Producer by stopping and clearing the DAQmx Task, and by putting a Sentinel, an easily-recognized value that won't ordinarily occur, on the Queue (as a signal for the Consumer).  In the case of the Queue being an Array of Waveforms, we put an Empty Array (which can never occur inside the loop) on the Queue as a Sentinel.

The Consumer gets the same Queue and dequeues elements.  Each Element is tested for the Sentinel (i.e. for an Empty Array).  If the test is false, then we have Data, which I here simply display on a Waveform Chart.  Otherwise, the True Case (not visible here) releases the Queue (as we can only get here if the Producer has exited, hence no more Queue elements will be put on the Queue, hence we no longer need the Queue) and the True value will stop the Consumer (after the Producer, as it should be).

Here's how it looks.  You should be able to re-create this in LabVIEW 2014 (don't worry about saving data to disk -- do something simple like showing the data on a Chart, much quicker and easier to see if it is working).

Bob Schor

[natasftw]

If you get that error, the memory you're worried about is the buffer on your DAQ device.

Essentially, you acquire data and then do something in your loop.  The "do something" takes enough time that you're not able to keep up with the acquisition rate you've chosen.

If you go to producer consumer, you split those tasks into two threads: get data and do something.

The do something loop starts to work in parallel.  If the do something takes longer then you're in a mess and need to figure it out.  But, it often resolves this issue.

[Yamaeda]

---

wrote:

I did some research and apparently a Producer/Consumer Template fixes this problem due to lack of memory. I don’t fully understand why there is a lack of memory. I had Task Manager open when I was running my VI and it did not show the 8GB of available memory being filled. So what exactly is the Producer/Consumer doing? I also am not exactly sure how to implement my current VI into the Producer/Consumer Template, as I have never used one before.

Attached is my VI.

Any advice would be appreciated!

---

As already mentioned your array will grow indefinatly, and you cannot use 8GB. An array requires continous memory and if running Lv 32 bit it can only access 4 GB to start with, so arrays over ~1GB will get problems.
The solution is simple, don't build the array continuously but write it to file instead. You can probably do it easily in the current design, but Producer/Consumer is always a good design choice.

/Y

[KurtzC1]

Hi Bob,

Thank you so much for the reply. I have a much better understanding of the producer consumer template now. I replicated the VI you attached in LabVIEW 2014. It runs and displays data, but I don’t know if it is working properly. How do you tell? I had the issue when I was storing data in memory, and I don’t think your VI is storing anything.

I then tried manipulating this VI so it writes to text file. I think the VI is writing over the data and not using a carriage return line feed. Therefore, I only get one line of data. Why is it not implementing the carriage return line feed? I am not 100% certain if this is the issue, but this is my first guess. LabVIEW also does not stop running my VI when I press the stop button, even though it appears to match your VI. So I am also not sure what is going on there. 

Sorry I did not reply sooner, I have been busy with midterms.

Attached are both VIs

Thank you again so much for your advice!

My VI which writes to a text fileYour VI which I copied

[KurtzC1]

I believe I did all of those things, and it still doesn't write data to a text file correctly. Any ideas? In another reply in this thread, is my updated VI with images and a deeper explanation of my issue. 

Thanks for the reply!