How to save data quickly at high sampling rate

[iyearning]

Hi all,

I am using a producer and customer structure for measuring and saving data. The sampling rate need to be as high as 10KHz. As there are so much data, it takes a long time before the data is saved. At first, I saved the data in excel sheet. Then I tried to save it in Binary file, but it still takes a long time to finish saving. How can I make the saving time shorting?

Thanks

best

[crossrulz · Accepted solution]

First let's do some fixes to your DAQmx code.  Since you are using Continous Sampling, do NOT wire up the Samples Per Channel.  That is actually limiting your buffer.  And there is really no need to set your buffer size either.  It defaults to really big, so that should not be an issue as long as you read your data often enough.

Now your data logging...You are just creating a huge array while acquiring the data.  Afterwards, you are saving the data.  This is actually bypassing the benefit of using the Producer-Consumer.  You should be saving your data in the consumer loop.  This will elinate the need for a lot of memory and you are saving data to the file while you acquire it.

But, in this case, I say the Producer-Consumer isn't even needed.  Use the DAQmx Configure Logging VI.  With that VI, you can tell DAQmx to stream all of the data directly to a TDMS file.  You don't have to do anything else.  This is by far the best way to save off your DAQ data.

[iyearning]

Hi crossrulz

I need to manipulate the date measured so it can not be directly saved using the way you recommend(Anyway, what you recommend is really fast!). I still need the producer-customer structure as I have very complex manipulation (like PID control loop). I attached a simple case to show my situation. I save the data to TDMS by using "write to measurement file". Was it the best/fast way to save data at high sensing rate?

Thanks

best

[deppSu · Accepted solution]

Yeah, TDMS is the best way to store measurement data and it also had some optimizations for storing DAQmx data.