Most efficient way of log and read data simultaneously at high data rates (DAQmx, TDMS)

[MarvSchl]

Hello,
 
I want to acquire data from several cDAQ Modules using multiple Chassis at
high data rates (100k samples per second if possible).Let's say the measurement time is 10minutes and we got a high number of Channels (40 for example). The measured data will be written into a TDMS file. I assume, the memory and/or the HDD speed are the limitations. For the user, there must be a possibility to display selection of Channels in a graph during the measurement.


My question: what is the best and most efficient way to log and read data simultaneously?

 
First of all, I'm using a producer-consumer architecture and I don't want to write and display the data in the same loop. I expect two possibilities:


[1] To use the "DAQmx configure logging.vi" with the operation "log and read" to write the data into a TDMS file. To display the data in a second loop, I would create a DVR (Data Value Reference) of the documented samples and "sent" the DVR to the second loop, where the data will be displayed in a graph. This method has the disadvantage that data of all channels will be copied in memory. Correct me when I am wrong.

 
[2] To use the "DAQmx configure logging.vi" but only with the operation "log" to write the data into a TDMS file. To display the selected data, I would read a defined number of samples from the TDMS file in the second loop (the TDMS file I'm currently writing to). In this case, I only have one copy of the data from selected channels (not from all channels), but there will be more HDD accesses necessary.


What is the most efficient and performant solution in this case?

Are there better ways to log and read data with high sample rates?

 

Thank you for your help.

[crossrulz]

Since you are also displaying the data, you want to use the DAQmx Configure Logging with "log and read".  Unless you have a lot of things happening (logic, math, etc) with your display data, I would just read it and write to the indicators in the same loop.  If you are getting something like 100ms worth of data per iteration, this will not be an issue at all.

[Blokk · Accepted solution]

You say the measurement time is 10 minutes. If you have 40 channels and you sample all CHs at 100 kHz, it is quite large number of values.

In such cases I always try to approach from the usage requirements. If one measurement will be only 10 minutes, I would just log all data to TDMS and create a Graph module which could be in the same consumer loop where you log the data. You can always work on the large raw data files offline later on, extracting all the info you need (have a look at the product called NI DIAdem: http://www.ni.com/diadem/)

The main question is what the user needs to see in the Graph (or maybe a Chart can be useful too). Lets say the Graph is 1024 pixel wide. It does not make any sense to show more data points than 1024, yes? Every second you will produce 100k data points per channel. What is the useful information what your user needs to see? This depends on the application. In similar cases I usually use a kind of data reduction method: I use a rolling average (Point by point Mean.VI for example) with an interval size of 100. In this way you get 100 1000 data points per channel at every second. If you feed your Chart at every second with these Mean values, it will be able to store (as default) 1024 data points per channel (curve), which is a bit more than 10 minutes, so the user will see the whole measurement.

So it all depends on how frequently you send data to the consumer. For example, you collect 1024 values per iteration of the Producer, and send this to the Consumer. Here you can do a normal Mean calc or a rolling one (depends on your requirements), and plot it to a Chart. In this way your Chart will show only the values of the last 10 seconds....

Once I programmed a kind of module where I use a Graph and not a chart, and the user can specify the interval by absolute time stamps which will be plotted. If the data size is bigger than the pixel size of the Graph, the modul does a Mean calculation to reduce the number of data points. Of course, if you need to see the raw data, you can specify an interval which is small. It all depends on how you program zoom functions, etc...In my case I hade only a 1 Hz rate, so I just kept all data in RAM limiting the arrays to keep 24 hours of data, so the technicians could monitor the system. In your case due to the huge amount of data, only a file write/read approach can work, if you really need to access all the RAW data on the fly. But hopefully the Rolling Mean values will be enough???

[MarvSchl]

Thank you for the detailed answer. It's a clever idea to reduce the data
points of the measurement equal to the pixel wide of the Graph with a mean
calculation. It is not necessary to access all the RAW data on the fly. To
display the data should be an information for the user what he is
currently measuring.

So in this case it's the best way to use DAQmx Configure Logging with the
"log and read" function to save the measurement data to a TDMS File and to
display selected data with a rolling mean.

When i want to display the raw data of selected channels or a small user defined
section of the data (for example minute 3-5) during the measurement, it's the best
to read out the values from the TDMS file in a separated loop on users request. Correct?

[Blokk · Accepted solution]

Anyway, if you only want to plot the reduced mean values on the Graph, it would be just much easier to configure the TDMS for only logging. When you calculate the mean values of the curves, these could be put into a cluster or a 2D array, and store them in a shift register. Of course you need to limit the max size of this array/cluster (one possible way is to check the growing array sizes, and remove older elements regularly). When the user want to read out specific intervals, you can take out the sub-array components for plotting.

You could do all the above functions (add data point(s), read out actual buffer at specified interval, etc) using a Functional Global Variable. You add the points to this shift register inside the FGV inside your Consumer loop. When the user wants to plot an interval, you just read out the values as needed.

[MarvSchl]

Thanks for the answer and for your help. I think, this is the solution.