High sample rate data acquisition using DAQ and saving data continuously. Also I would like to chunck data into a new file in every 32M

[lawsberry_pi]

Hi: 

  I am very new to LabView, so I need some help to come up with an idea that can help me save data continuously in real time. Also I don't want the file to be too big, so I would like to crete a new file in every 32 mega bytes, and clear the previous buffer. Now I have this code can save voltage data to TDMS file, and the sample rate is 2m Hz, so the volume of data increase very fast, and my computer only have 2G ram, so the computer will freeze after 10 seconds I start to collect data. I need some advise from you briliant people.

Thanks very much I really appreciate that. 

[Bob_Schor]

As long as you can write to disk at a faster rate than you acquire the data, you can (usually) use the parallel nature of LabVIEW to set up a Producer/Consumer Design Pattern to "stream" your acquired data directly to disk.  There are numerous examples in LabVIEW (including Example Projects and New Templates), but this is the general idea:

• Have a Producer Loop that runs at some rate (say N points at a time).  Put these points into a Queue (which gets them out of the Producer loop and lets the loop keep running).
• Bring the Queue to the Consumer loop, which dequeues the N points and writes them to an already-opened file and continues as long as points show up in the Queue.
• Create the Queue with enough space to accomodate minor fluctuations in the rate that the Consumer loop runs (due to unpredictable times for moving disk heads and platters, for instance).

The idea is that your Producer loop acquires data and immediately hands it off to the Queue.  Perhaps the Consumer isn't ready to empty the Queue, so a dozen buffers-full of data pile up in the Queue.  But then the disk I/O is ready and the Consumer loop starts emptying in the Queue by writing the data to disk.  As long as disk writing (on average) is faster than data acquisition, the Queue size will always be driven to 0.  Plus at any point in time, you only need to keep (in memory) the dozen or so buffer space for the Queue (to handle the speed variations between the regularly-timed Producer Loop and the non-deterministic Consumer Loop).

Bob Schor

P.S. -- with this Producer/Consumer model, you might not need to worry about "chunking" data into multiple files, which will simplify your code and processing.

[crossrulz · Accepted solution]

I am a huge proponent of the Producer/Consumer architecture.  But this is the place I advise against it.  The DAQmx Configure Logging does all of it for you!

Note: You will want to use a Chart instead of a graph here.

[Bob_Schor]

Crossrulz,

     As (almost) always, I agree with you.  I was addressing the second part of his question, "chunking" data to disk.  Wouldn't you agree that this latter task should be handled with a Producer/Consumer design?

BS

[Bob_Schor]

Aha, I didn't notice the Log and Read setting in your DAQmx code ...  Oops ...

BS

[Hooovahh]

---

@Bob_Schor wrote:

Aha, I didn't notice the Log and Read setting in your DAQmx code ...  Oops ...

---

Yeah that's a neat little feature added to the DAQmx TDMS logging functions around the 2010 era.

[crossrulz]

---

@Hooovahh wrote:

---

@Bob_Schor wrote:

Aha, I didn't notice the Log and Read setting in your DAQmx code ...  Oops ...

---

Yeah that's a neat little feature added to the DAQmx TDMS logging functions around the 2010 era.

---

Specifically DAQmx 9.0 (which I think was released with LabVIEW 2009).  That feature saved me MONTHS of development time I was about to do on a data logger.  Within a day (I played around with it more than I should have) I had a task I was given 3 months of hours to do done.

[lawsberry_pi]

Thanks very much! I try your code, and it's working very well. The method is so useful and easy! I can't believe that I don't event need to creat a tdms file!

Thanks very much guys 

[Bob_Schor]

One of the best parts of the Forums is that even Know-It-Alls (I won't look in the mirror!) can learn new things!

Bob (Humble) Schor

[crossrulz]

---

@Bob_Schor wrote:

One of the best parts of the Forums is that even Know-It-Alls (I won't look in the mirror!) can learn new things!

---

Indeed.  I still learn things from the forums, if not from other users, then by trying to figure out the answers.