producer consumer memory overflow

[Trehcir]

I am trying to log data from 4 ports at 40.96Mhz. using a producer consumer architecture and a NI PCIe 6527B using an external clock.

I inherited code that works for 2 ports.  I tried to add the additional ports and changed the data size for the DAQ.mx and data queue.  Running the program several times, I reach conflicting error messages after less than 2 seconds from start:

Error 200284: Some or all of requested samples have not yet been acquired.  [...] To make samples available sooner, increase your sample rate

Error 200361: Onboard device memory overflow.  Because of system and/or bus-bandwidth limitations, the driver could not read from the device fast enough to keep up with the device throughput.  Reduce your sample rate.

Is this a race condition between error messages, a start up timing issue, or something else?

How do I approach debugging these types of problems?  How do I determine what is triggering the error?  I can't single step through because I'm driving sampling with an external clock.

Running on labview 2012 on Windows XP on machine that has 3.17GB ram.

[Bob_Schor]

I'm "on the road" and can't try things out, but I'm wondering if you are having Queue allocation and timing problems.  As I look at you code, you seem to be taking 2,046,000 (should that, possibly, be 2,048,000?) samples from 4 channels, or roughly a 4 by 2 million array of U32 (why U32?  Aren't you sampling from lines 0 .. 7, which fit in a byte?).  So each time you generate a sample, you need to gather 32MB of Queue space, which hasn't been allocated.  Note that you specified a Queue of unspecified size and shape, and of unspecified length, so in addition to everything else, LabVIEW has to juggle memory management.

Since this is on a PCIe board, I'm assuming you are doing all of this in Windows, which is not a Real-Time OS, hence not deterministic, so it's a minor miracle that it works for fewer channels.

Look up (I think you might find it in the Help files) suggestions for pre-allocating Queue space.  You can definitely speed up Pro/Cons designs by setting the Queue up prior to using it.

Bob Schor

[crossrulz · Accepted solution]

Try reducing your buffer size.  In fact, just leave that unwired in the DAQmx Timing.  The buffer size should be plenty large by default.  And then also try reading less data at a time, like 10ms of data instead of 500ms of data.  What I am suspecting here is that the DAQmx buffer is too large and therefore must use the harddrive for part of the buffer, slowing things down a lot.

Other notes here:

1. Do not use the timeout to stop your consumer loop.  Your producer should send a command of some type to your consumer telling it to stop.  In this case, I would use an empty array.  And then you should release the queue after the consumer loop stops.

2. You will want to add a FALSE constant to the Append Array/String Size input of the Write Binary File.  This will remove the 2 I32s at the beginning of each array that you write.  Tthey are there to tell the size of the array being written.  But since you are just streaming data, I doubt you care about those.

3. Instead of the whole Producer/Consumer setup, have you considered just using the DAQmx Configure Logging VI to let DAQmx stream the data straight to a TDMS file for you?  Makes things a lot faster and a lot easier on you.

[Trehcir]

Thank you, I will look into that.

I am using u32 because I am reading from 4 8-bit ports (4*8=32).  I only use 16 of the bits, but the lines are on different ports and I don't think I can select individual bits to read.

The DAQ.mx, as I understand it, is greedy and reads as much data as it can because it is reading continuously, so it is not necessarily reading all the 2.092*10^6 samples, but I think you are right that this is a timing and memory allocation problem, and that the queue is not keeping up. 

You are also right that I am running on Windows, unfortunately.

[crossrulz]

---

Trehcir wrote:

The DAQ.mx, as I understand it, is greedy and reads as much data as it can because it is reading continuously, so it is not necessarily reading all the 2.092*10^6 samples, but I think you are right that this is a timing and memory allocation problem, and that the queue is not keeping up.

---

You told the DAQmx Read to read 2.092*10^6 samples, so it will try to get than many samples in a single read.  Your errors do not point to a queue error, but a DAQmx setup error.

[Trehcir]

I haven't gotten it working yet because I'm not experienced with labview, but I think the DAQ.mx configure logging block is meant for this type of problem, and I just need to fiddle with it a bit more to make it work to log continuously.