How to log data in different rates in a single VI?

[Hao_Liu]

Hi, 

i tried to acquire 2 channel of voltage measurement at 10KHz and then analyze the data, finally record the data at both 1KHz and 1Hz. What i have done is that use producer/consumer loop to achieve the recording work at different rates. I enqueued all the data in the main loop, and then dequeued the data and write them into file in other loops. I added 'wait=1 ms' in one consumer loop to achieve data logging at 1KHz (fast logging loop), and added 'wait=1000ms' in another loop to achieve data logging at 1Hz (slow logging loop). 

My problem is that the data logging works really well in the 1Hz case but it doesnt work for 1KHz case. Although the wait time set as 1ms, the interval between two samples in the fast data logging loop is around 100ms.  Afterwards, i tried 1ms, 10ms, 100ms,150ms in this loop to see what the intervals are in those cases. I found that the intervals are same as assigned in the wait time greater than 100ms, but if the wait time is set as 1ms or 10ms, the intervals are again around 100ms. 

My question is that why i can not achieve 1KHz data logging by using consumer/producer loop whilst the same way works for 1Hz case? 

Many Thanks,

Hao 

[crossrulz]

You shouldn't have any waits in your consumer loops.  They should just dequeue and do whatever is needed with the data.  Can you share some code so we can see what you are doing?

[Bob_Schor · Accepted solution]

You are sampling at 10KHz and want to save your data "as though you sampled it at 1KHz and 1Hz", as I understand it.  I also understand that you are using the Producer/Consumer Design Pattern to help with this (a Very Good Idea).

I'm assuming that the Producer "produces" a chunk of data at a time, say a second's worth of data, or a 2 x 10000 I16 (assuming a 16-bit integer A/D) array, which you put in a Queue and dequeue in the Consumer.

If you have data at 10KHz, how do you get representations at 1KHz or 1Hz?  There are (at least) two possibilities -- average the data over the appropriate time interval, or (re)sample the data.  The nice thing is that you can handle both the 1KHz and 1Hz situation basically at the same time, and in the same processing loop.  I'll discuss the 1KHz case -- with the assumptions I made, the 1Hz case becomes almost trivial.  I'll also just discuss processing one channel, as it is similarly almost trivial to go from 1 to N channels.

So you have 1 second of data sampled at 10KHz, and you want to "reduce" it to 1 second of data sampled at 1KHz.  Note that you have 10 samples for each data point that you ultimately want.  The "Averaging" method says "Replace every 10 points with the Average of those 10 points", while the "Resampling" method says "Replace 10 points with, say, the first point, ignoring the others".  One easy way to do this is to use the Reshape function to turn your 1-D 10000-element array into a 2-D 10x1000 element array.  Now you can use a FOR loop to either average every 10 elements or simply extract the first.  [Here's where almost everyone, including me, makes a mistake -- when you put a 10x1000 2-D array into a FOR loop, you get 10 iterations of 1000 elements, and we want 1000 iterations of 10 elements.  Oops, what to do?  Simple -- transpose the 2-D array before sending it into the FOR loop.]

So when you've processed your 10,000 points and gotten both 1000 "1KHz" samples and 1 "1Hz" sample, you write each of them to their respective file, and wait for the next element to appear in the Consumer Queue.  Since the Consumer Loop is Queue-driven, it will automatically "block", waiting for the Producer to "feed it" some data.  Note that the Producer generates data once per second, producing 10000 points for the Consumer to process, but the time to process these points and do the required Disk I/O should only take a few milliseconds (certainly no more than 100, even if writing to a floppy disk!), so you'll have plenty of time to do other interesting things (like display the data).

Bob Schor

[Hao_Liu]

Hi,

Thank you very much, i think my problem is solved now. You mentioned that the consumer/producer loop is a queue-driven. If my producer loop has other calculations(say FFT of the acquired waveform) based on the acquired data, is it possible that the calculations slow down the whole loop and the queue is also accumulated slowly?

Many Thanks

Hao

[Bob_Schor]

The idea of a Producer/Consumer loop is that the Producer (which is the time-driven, time-critical part that produces the data) does almost nothing except "produce" data -- it does no FFT, no processing, so all of its cycles are devoted to doing the time-critical task.  The Consumer does all of the processing, including FFT and the like.  Note that there is nothing wrong with having a Consumer also act as a Producer.

For example, you acquire data in a timed loop, and send it via a Queue to the Consumer loop.  The Consumer wants to do two things -- save all of the data to disk (and doesn't want to "miss" any of the data) and also process data (FFT, etc.) and display it for the user.  If too much data comes in to keep up with the display, it's OK if some data are "lost" and not displayed, as you can always go back and analyze the data saved to disk.

So what you'd do in this case is make a Consumer loop that accepts data (via a Queue) from the Producer.  It would do the following -- write out the data to disk, and place the data on an "Analysis/Display" Queue for "consumption" by a third, Analyze/Display loop.  Note that this Consumer should have no problems keeping pace with the Data Acquisition loop (as it is doing very little processing).

Now the Queue that you use for sending the data to the Analyze/Display loop should be configured a little differently, particularly if you think that the third loop might not be fast enough to keep up with all of the data.  What you don't want is an "unbounded" Queue, one whose size is the default of -1 (which just keeps expanding until you run out of memory).  Instead, decide on an upper limit of the number of elements you can hold (say, 100) and use the "Lossy Enqueue Element" method to put new elements on the Queue.  What this does is to prevent the Queue from "blocking" because it is full (which would stop you middle loop, trying to save the data to disk, and you'd potentially lose data!) at the cost of discarding the oldest element in the Queue.  In practical terms, this means that if you can't analyze/display the data as fast as it comes in, the Queue will take care of deciding which points to omit (as they fall off the end of the Queue).

Ultimately, such a tiered Queuing system keeps your important loops running, your data flowing, and maximizes the power of LabVIEW to seamlessly run multiple loops in parallel, letting you prioritize (in a natural way) where the processing power should be allocated.

Bob Schor

[Hao_Liu]

Thank you very much. As a beginner,  i tried to build everything in a single loop to make it somehow 'look simple'. What you explained is really good and clear. I will use multiple loops to run different tasks. 

Hao Liu