Problem working with large data ...

[vjuno]

Hello Everybody,

We are using two PCI 6120 cards to acquire data from 6 channels (3 on each) .  I am writing a VB .NET program for the same. We want to acquire data at the rate or 1MHz, for around 10 minutes. And my experiment demands that I do not lose any data.  I derived my code from the example programs available with DAQmx and MStudio. In my program, data is collected at 1MHz, 10^5 points at a time. Then these  3*10^5 points (right now I am using only one card for three channels) are saved as arraylist (as done in the example program). And then at the end of acqusition (it is only 10 secs currently), data is written to the file. But it takes very long to write the data to file.  I wan to write data into the file, simultaneously it is being acquired. I tried to write data to file immediately after the read operation. However, after a few acquisitions I see the error that Task has been stopped to prevent overwriting of the databuffer. Can somebody help me in this regard please? I am using the properties 'DoNotOverwrtie UnreadSamples and ReadCurrentPosition' for my tasks.

 Thank you in anticipation.

Regards

Vijay 

[Jordan F]

It is going to be near impossible to stream to a file at a rate of 1MHz without overflowing the buffer on your PCI-6120.  However, there are ways to possibly get around this.  One way that you can avoid the buffer overflow is to have separate the acquisition and writing to file.  I know you are not using LabVIEW but this article on Producer/Consumer architecture might be helpful in understanding this concept.  Producer Consumer uses queues to store the data and then in a separate loop write the data to the file.

Application Design Patterns: Producer/Consumer

Another idea would be to have multiple buffers that store the data in computer memory and then take that stored data and write it to a file.  This could get pretty complicated though and would require effort in knowing what data is being written to the file and the position of where you need to read from in the buffer.

[drohacek]

Vijay -

Please see this thread - http://forums.ni.com/ni/board/message?board.id=232&message.id=2571&query.id=64680#M2571

for an example of using a .NET Queue to quickly store the data being acquired into memory. In your case, you'll want the thread that is pulling the data off of the Queue to write it to disk instead of displaying it on the UI.

David Rohacek

National Instruments