Data Acquisition At High Rate in labview

[ritntu]

Hello Everyone. I am Using NI-USRP 2954 for some experiment and my received signal is received at a rate of 2.5MS/s. I need to keep on collect the data for atleast few second. I am using "write to measurement file" vi from File I/O group for the data logging. However when I am collecting data it is gathering data very slow. There becomes a huge difference between number of transmitted and received chirp. If I am not collecting the data then there is no difference between the transmitted and received chirp, i.e., if i am transmitting 10 chirp in a duration of time i am getting 10 chirps at receiver. So, what is the way to to acquire the data in such a high rate. 

[GerdW]

Hi ritntu,

---

@ritntu wrote:

So, what is the way to to acquire the data in such a high rate.

Don't use ExpressVIs when it comes to high-speed requirements!

Use basic file functions and (most important) open the file only once before the loop and close it once after the loop.

You may even use a producer-consumer scheme to decouple DAQ from file operations...

[jjohn1]

I have had a few applications where I was trying to maximize streaming rate from a digitizer (NI-scope mostly).

If you have the option to transfer and store the raw unscaled integer data this might help.  There is no reason to store 64 bit floating point numbers for data that is at best 16 bits.  Just save the raw data and scaling factors.

Also, in my tests producer/consumer loops actually supported a higher streaming rate than reading and writing in the same loop.  I am not sure why, but it may have something to do with data copies when buffering the writes to disk.

[RTSLVU]

You should be using a Producer/Consumer architecture for any type of high speed data acquisition. Especially of you plan on analyzing, visualizing or saving live data at the same time.

In a nutshell a Producer/Consumer has two parallel loops connected by a Queue or Channel Wire. The Producer loop handles the data acquisition and puts the acquired data into the Queue allowing it to run full speed. The Consumer loop can then take it's own time to dequeue, analyze, display, decode, and save the data without slowing down the data acquisition or losing any data. 

[ritntu]

Hello RTSLUV. Thank you for the suggestion. Can you provide some good sources to learn Producer/Consumer architecture. If there is any existing vi this will be more helpful. My output data is a array of complex value. As you can see the image i am extracting real and imaginary part before saving it. I hope in my case also this architecture will work.  

[RTSLVU]

Here's a good place to start Producer/Consumer Architecture in LabVIEW

There's plenty of other examples and tutorials that you can find with Google.

[mcduff]

There is some good advice in the comments, just to add:

1. Use low level File/TDMS functions for saving data.
2. Save the unscaled binary data, not the scaled data.
3. A few seconds of 2.5MSa/s is not a huge data stream. (I have been able to stream 4 16 bit channels at 125MSa/s each channel to a RAID array continuously until storage ran out.) Here a some options.
   1. Store the received data in array, then save the data to the disk. I assume you have gigabytes of RAM. Five seconds of 16 bit I&Q data is 50MBytes, not a huge value with respect to gigabytes.
   2. Use the Producer/Consumer as recommended. Here is the important part to increase disk writing speed. Write in even multiples of the disk sector size.  Typically the disk sector size is 512 By, so the number of points you write should be an even multiple of 512.
4. Learn to use TDMS file functions, it is a file format meant for streaming.

[ritntu]

Thank you mcduff for the suggestion. While I am using the tdms for collecting the data, it is getting saved as tdm extension instead of tdms. In matlab there is no way to read tdm. how can i save directly to tdms extension file.

[GerdW · Accepted solution]

Hi ritntu,

---

@ritntu wrote:

While I am using the tdms for collecting the data, it is getting saved as tdm extension instead of tdms. … how can i save directly to tdms extension file.

---

Use the TDMS file functions to create TDMS files…

[ritntu]

Yes i used tdms file functions, but it is saving data with .tdm function not iwith .tdms.