Passing Data with Many Senders at Multiple Rates for Logging

[Nathan_S]

The image above shows my situation.  I have a 10Hz Datasocket Read, a 10Hz UDP Read, and a 100Hz DAQmx Read.  I need to assemble all this data into a single stream for logging.  If I have waits for my network stuff, I won't be able to assemble the data (due to the data flow dependencies).  If I set timeouts (10 ms for 100Hz), the UDP Read will throw an error.  Then if I ignore the error, I will never know if the network actually times out.  Any help is appreciated.

The code is obviously here for representation and not meant to be a working piece of code

[Nathan_S]

Bump

Any help?

[Ray.R]

Have you considered using a Queue?

[Nathan_S]

Yeah, but my only concern with using a Queue is how to mesh the data from different locations into a single stream.  I was thinking using a 1-D array of waveforms, so I could just bring in the data by index.  Also, I'm not sure what will happen if all the loops don't mesh their data well (especially coming from two network sources that might experience a small amount of latency).  I was thinking either Queue or DVR (maybe? didn't seem to promising due to the serialized nature of DVR interface) or...?

[tst]

---

@Nathan_S wrote:

Yeah, but my only concern with using a Queue is how to mesh the data from different locations into a single stream.

---

You obtain the reference to the queue once and then use that reference in each of the loops to enqueue the data. Each data will go into the queue when it arrives, which will sort them in their order of arrival, which sounds exactly like what you want.

[Nathan_S]

I agree about the approach, I guess I worded that phrase poorly.  Let me try to explain it better.

DAQmx data - 100Hz Channels N->X-1

Datasocket data - 10 Hz Channels X->Y-1

UDP data - 10Hz Channels Y-Z

I can pass a reference to the queue to each loop and queue the data as an array of waveforms.  I think I can use an index approach to keep the data sorted.  But what if:

DAQmx data - fires 3 times (aka 3 seconds' worth of data).

Datasocket - fires 1 time (due to network lag)

UDP - fires 3 times (everything works perfectly)

What happens to my stream of data here if the above situation occurs?  It seems like logging would not respond well.  I was thinking instead of using a cluster of queues so that each piece of data could be passed individually.  This would make it so that I would never have the problems like I'm talking about above.  I think this would be a viable method.  Any opinions?

[apok]

did you look at this in the find example>>Queue Multiplexer.vi....."Many-Into-One"

[tst]

I still can't say I understand the problem. Do you mean that you're afraid that the data will come in late because of various potential lags, but you want to actually sort it by the value of the timestamp inside the waveform?

If that's the case, then I would suggest either logging all the data in the queue and then flushing it all out at the end and sorting it or (if that's not acceptable because it will be too much) using two alternating queues where each queue logs data for N seconds (a length of time which you trust will be longer then your lags) and when you switch to the second queue you flush the first, sort its data and then enqueue the sorted data into a third queue, which is where your final data will live.

[JÞB]

If the queue data is a cluster of timestamp source and data you can write the data to the correct channel in the tdms file as it arrives by simply selecting the channel to write by the source.

[Ray.R]

I can't add much more to what has already been said.  What they proposed is how I would implement it.

Hence, why I was suggesting the use of a Queue.  🙂