Best way to save data from different data-sources, timestamps and a changing samplerate --> to one file via Buffer

[Tanchris]

Hello,

I have a named queue for multiple data-sources with different samplerates. I buffer them in a circular buffer for example 10sec. All tasks start via rendezvous and I got the start-measure-timestamp. So far so good.

During the measurement the samplerate of a step in a sequence could  change (for example one step is 10s with 1kHz and the next one 2s with 10Hz, not the samplerate from the data-source itself).

But now I want extract the correct data from the circular buffer and I´m looking for the best way.

At the end the data should be saved in a tdms-file for the correct steptime und resampled samplerate and all data-sources.

I want it make like this (but too many open questions, how to implement):

-> got start-timestamp

-> looking into the buffer and extract the last second of measurement (so it means start-timestamp+1s) if this data are already available

-> looking, is the steptime elapsed? If not, resample all data to the requested step-samplerate

-> ???

What is your best way, to save data from multiple sources (and all sources could have different samplerates), then resample the data to the requested samplerate per step an a sequence?

The continuous measurement and logging-example is only for one source. And other example I didn`t find...

[Kevin_Price]

I'm pretty confused by your description, but I *think* you're in a spot where you started off in the wrong direction, and are looking for advice on how you can continue in that wrong direction and still get where you're headed.  My advice is to back up to the beginning and get started in the right direction in the first place.

   I've done quite a few apps where one central consumer receives data from multiple data producers at different rates.   My queue datatype will typically be a typedef'ed cluster consisting of a string and a variant, just like the QMH template that comes with LabVIEW.

   The string is used to identify the data source, and then the variant can carry the data no matter what "shape" the data is.  The string-variant pair is considered a "message" in the QMH pattern, and my consumer loop will respond differently to the distinct messages.

   When I receive one with string="Force", variant=1D array of waveforms, the code inside the "Force" case of the case structure knows what to do with the force data (starting with converting from variant back to 1D array of waveforms).  When I receive one with string="Position", variant=1D array of DBL, the code inside the "Position" case similarly knows what to do with the position data.   And so on.

   Repeating: all data sources push data into the same queue using the very flexible queue datatype of a string-variant pair.  The consumer can now easily do distinct and unique handling for each of the data sources, as needed.

   I usually don't do file logging in that main consumer loop because file operations can sometimes be slow to execute.  I have a separate loop dedicated to file logging.  Depending on the app, I may have data sources send a copy of all their raw data there directly, I may have my consumer loop process the data in some manner and then send it off for logging, and sometimes I do both.

Hope I understood your question enough that this'll get you headed in a better direction.

-Kevin P

[Tanchris]

Probably you´re right. I had a stable system, but know the circumstances has changed.

I do it like you. I got all data in a buffer. This buffer has a cluster for every device and his channels. Its a cluster with the device-type and all the waveforms.

The "data-collection" is one separate process and actually the resampling another one (coupled via queue, but separate low coupled VI´s).

My real problem now is: how to separate the data from the buffer...the buffer works fine. But for example one problem is: how to separate the data by timestamp...searching the waveforms? But perhaps slow data didn`t have the timestamp to and its nescasary to search next to them. Or next problem: should I request the timesstamp if the next step is starting and take the data from this point? Seems so.

I´m sure, a lot of people have this problem sometime - I would like to know, how these people solve this. So that is the best way --> if you have a sequence with steps. These steps have different samplerates. The data came from multiple sources (and they have different samplerates to). And at the end the data should saved in tdms and all sources are resampled correct.

Yes, I´m it looks like that I must change something in my programm...but thats not the problem...but I want do in the best way 🙂