high speed daq streaming with queue

[whitsy]

Hello,

I am hoping the experts here can help with my problem.  I am trying to stream high speed (1-2 MSPS) DAQ data to disk.  I want to utilize a queue and 2 parallel loops, one to read the data from the DAQ and put it in the queue, and the other to pull from the queue and write to disk. 

In my first test I am able to do that when I use a single double as the queue element.  I run the code with a sine wave input and can extract that from the .bin file of doubles that I am streaming to.  This does not support the data rates that I need, so I want to use and array as the queue element and then read larger chunks of data from the DAQ.  It appears to me that the data is read correctly,  but when I read the data from the file it looks like it is missing samples.  Any help/suggestions is appreciated.  I will post my code.  Thanks

[GerdW · Accepted solution]

Hi whitsy,

why don't you use the built-in feature of DAQmx to stream the DAQ data directly into a TDMS file?

On your VI:

When using WriteToBinaryFile you use the default setting of including the array size within the data. Maybe this is disturbing you when reading the data?

[whitsy]

GerdW,

I did not know that this TDMS streaming was a thing, or now that I know it is a thing what the performance is.  As I had stated earlier I will be trying to stream up to 2 MBPS to disk.  

There is a 2nd reason why I wanted to try the queue.  At a later time I may be adding some kind of signal processing to the data in real time and I wanted to utilize the 4 cores of my computer for this.

I will look into the TDMS for my immediate concern, but I was looking to see if anybody saw anything glaring in my vi as to why my logged data was wrong.  Thanks

[crossrulz · Accepted solution]

The Stream To TDMS functionality in DAQmx is more efficient than a Producer/Consumer.  It avoids layers of software to get the data to a file, making it efficient.  Granted, it might still struggle with the data rates you need, but it will be better.

The only thing I saw in your code as far as structure was the wait in the consumer loop.  You typically want your consumers to run as fast as possible.  If they get ahead of the producers, then they will sleep waiting for data to dequeue.  So very rarely should you have a wait in the consumer loop.

[whitsy]

Gerdw,

I change me .vi to write to a tdms file. Everything is working great. Thanks, whits.