Saving Data in Separate Locations using Multiple Queues

[nikvl]

Hi all,

I'm acquiring data at a certain interval (100ms) and saving it at a slower rate (500ms) by enqueuing after a certain elapsed time. This data is dequeued to be saved in local text files. The same data should also be saved to a databank on a server. If I were to combine save data locally and on the server in the same loop, this process slows the loop down considerably as I believe there's some latency in trying to access the databank. I would also like to separate loops so that if connection to the databank is severed, the program continues running and saving to the local file. The reason data is enqueued again into a new queue after dequeing is because you can only dequeue once and to ensure that the data saved in the databank is identical to the one in local file.

The attached vi is a simplified form but it's rather rube-goldberg-esque and perhaps there are more elegant ways to save acquired data in two separate places independently.

Thank you in advance!

[wiebe@CARYA]

Queues are most practical for many to 1 solutions.

If you dequeue an element from a queue at one location, the element is gone from the queue.

I'd suggest user events, or channel wires. Or 1 queue for each receiver.

[TiTou]

If the amount of data is small that's fine, if it's large, I'd suggest to place a map in a DVR, key of the map being the index of the packet, as you have a fixed acquisition rate, it's easy for multiple loops to access the map in the DVR and decide if saving in needed by doing a modulo on the index.

The acquisition loop has to manage the size of the map in the DVR.

Again, this would be overkill if the data in memory is small.

[Bob_Schor]

You tell us how often you acquire data (every 100 ms), but not your sampling rate (or, equivalently, how many "points" are in those 100 ms).

Assuming your sampling rate is on the order of kHz (instead of MHz), you should have no problem saving the data (from a single "source", like a DAQmx Acquisition Loop) in two separate data repositories (call them "Local" and "Server").  What I would recommend is to have two "sequential" Queues, and three independent Loops.

The main Acquisition Loop puts each "acquisition data packet" into a Queue (which I'll call the "Local Queue").  Because you are generating data packets every 100 ms (or at 10 Hz), you'll be putting data into the Local Queue at 10 Hz.  Make your Queues have unlimited length.  Note that your data generating loop (which also runs at 10 Hz) is basically idle, waiting for a chunk of data (maybe a few hundred data points) to arrive every 100 msec, so there is plenty of time for other loops to run.

So the Local Queue goes to the "Save Local" parallel loop, which does the following:

1. It dequeues the data from Local Queue, and Enqueues it in a second Queue, "Server Queue".  Notice this is just a "data copy", so takes almost no time (compared to the 100 ms we have between Queue Arrivals).
2. It writes the data (which is still sitting on the wire going from the Dequeue from Local Queue to Enqueue to Server Queue) into Local File.  Even if it has to open the file, this will take only a second or two, which may create a small backup in the Local Queue, but you've got a Queue of "unlimited size", which is certainly > a few seconds of Enqueue/Dequeues.
3. The Server Queue loop dequeues the Server Queue and writes the data packet into the Server File.  The argument that you can spend a few Queue "cycles" opening the data file in the Server, and possibly handling data traffic to the Server, should also be relatively trivial.

An important element of working with Queues (especially "sequential Queues") is how to shut them down.  The same way that Queues are created from "Producer" to "Consumer", the proper way to shut down a Queue is to pass a "sentinel" from the Producer to the Consumer.  In this case, the Queue elements are arrays of data samples (for example, your DAQ loop might consist of 4 channels of 100 samples collected at 1 kHz).  The Producer (the first loop) "knows" when you stop collecting data (and stop the main loop from running).  Bring the Queue out of the Producer loop, but do not dispose of the Queue.  Instead, enqueue one more data packet, sending an empty data array (in the example I'm describing, it would be an empty 2D array).  In the Consumer, when you dequeue the data, you need a Case statement where you test for an empty Array.  Your "Consumer" code goes in the "False" case (where you save it on the Local File and enqueue it to the Server Loop).  In the True case, you "know" (because the Producer "told you" it was done) that you can safely dispose of the incoming Queue, and can create a similar Sentinel (an empty array) to send to the Server Loop.  You can also close the Local File and exit the Local Loop.  Needless to say, you do the same thing in the Server Loop -- test for the Sentinel, and if so, Dispose of the Queue (and close the Server File) and exit the Server Loop.

Would I do it this way?  Well, when I had to do something like this in LabVIEW 7, that's just what I did.  But in LabVIEW 2016, NI released Asynchronous Channel Wires, so I would use Stream Channels (which have build-in Sentinels and have "channel wires" that run in the more logical "left-to-right" direction so that the Stream goes out the right side of the enclosing While Loop.  You can see this in action by looking at the "Channel - Stream" example that ships with LabVIEW.

Bob Schor

[nikvl]

---

@Bob_Schor wrote:

You tell us how often you acquire data (every 100 ms), but not your sampling rate (or, equivalently, how many "points" are in those 100 ms).

Assuming your sampling rate is on the order of kHz (instead of MHz), you should have no problem saving the data (from a single "source", like a DAQmx Acquisition Loop) in two separate data repositories (call them "Local" and "Server").  What I would recommend is to have two "sequential" Queues, and three independent Loops.

The main Acquisition Loop puts each "acquisition data packet" into a Queue (which I'll call the "Local Queue").  Because you are generating data packets every 100 ms (or at 10 Hz), you'll be putting data into the Local Queue at 10 Hz.  Make your Queues have unlimited length.  Note that your data generating loop (which also runs at 10 Hz) is basically idle, waiting for a chunk of data (maybe a few hundred data points) to arrive every 100 msec, so there is plenty of time for other loops to run.

So the Local Queue goes to the "Save Local" parallel loop, which does the following:

1. It dequeues the data from Local Queue, and Enqueues it in a second Queue, "Server Queue".  Notice this is just a "data copy", so takes almost no time (compared to the 100 ms we have between Queue Arrivals).
2. It writes the data (which is still sitting on the wire going from the Dequeue from Local Queue to Enqueue to Server Queue) into Local File.  Even if it has to open the file, this will take only a second or two, which may create a small backup in the Local Queue, but you've got a Queue of "unlimited size", which is certainly > a few seconds of Enqueue/Dequeues.
3. The Server Queue loop dequeues the Server Queue and writes the data packet into the Server File.  The argument that you can spend a few Queue "cycles" opening the data file in the Server, and possibly handling data traffic to the Server, should also be relatively trivial.

An important element of working with Queues (especially "sequential Queues") is how to shut them down.  The same way that Queues are created from "Producer" to "Consumer", the proper way to shut down a Queue is to pass a "sentinel" from the Producer to the Consumer.  In this case, the Queue elements are arrays of data samples (for example, your DAQ loop might consist of 4 channels of 100 samples collected at 1 kHz).  The Producer (the first loop) "knows" when you stop collecting data (and stop the main loop from running).  Bring the Queue out of the Producer loop, but do not dispose of the Queue.  Instead, enqueue one more data packet, sending an empty data array (in the example I'm describing, it would be an empty 2D array).  In the Consumer, when you dequeue the data, you need a Case statement where you test for an empty Array.  Your "Consumer" code goes in the "False" case (where you save it on the Local File and enqueue it to the Server Loop).  In the True case, you "know" (because the Producer "told you" it was done) that you can safely dispose of the incoming Queue, and can create a similar Sentinel (an empty array) to send to the Server Loop.  You can also close the Local File and exit the Local Loop.  Needless to say, you do the same thing in the Server Loop -- test for the Sentinel, and if so, Dispose of the Queue (and close the Server File) and exit the Server Loop.

Would I do it this way?  Well, when I had to do something like this in LabVIEW 7, that's just what I did.  But in LabVIEW 2016, NI released Asynchronous Channel Wires, so I would use Stream Channels (which have build-in Sentinels and have "channel wires" that run in the more logical "left-to-right" direction so that the Stream goes out the right side of the enclosing While Loop.  You can see this in action by looking at the "Channel - Stream" example that ships with LabVIEW.

Bob Schor

---

Hi, thanks for the reply. If you opened my vi that is what I did, with three separate loops and sequential queuing. I guess my question was whether it's fine to dequeue and then enqueue once again. I understand what you mean by exiting the loops with the empty array but the way the program is built is that I don't necessarily want to exit the loop after dequeue is done, because the user could press a button to restart the process.

I also came across Channel Wires while searching for ways for this problem but since I've never used them before I wasn't sure if I should implement it at this stage. What is the main difference from a Channel Wire to a queue? Is it 'faster'? What would be the disadvantage/dangers of using channel wires instead of queues? Would I have to create two streams from data acquisition to both 'Local' and 'Server' or do it sequentially like the queue?

[wiebe@CARYA]

---

@nikvl wrote:

What is the main difference from a Channel Wire to a queue? Is it 'faster'? What would be the disadvantage/dangers of using channel wires instead of queues? Would I have to create two streams from data acquisition to both 'Local' and 'Server' or do it sequentially like the queue?

---

There's no performance benefit in using a channel wire. Channel wires are just abstraction layers build with queues and other standard functions.

Channel wires can be convenient though. They can take way a lot of boiler plate work. So development can be faster.

If you choose the correct channel wire, overhead will be small to minimal.

There are a bunch of them, and I hardly used any of them. The times I did use one (mostly for quick examples), they did deliver and where fast to develop.

[Bob_Schor]

---

@nikvl wrote: Hi, thanks for the reply. If you opened my vi that is what I did, with three separate loops and sequential queuing. I guess my question was whether it's fine to dequeue and then enqueue once again. I understand what you mean by exiting the loops with the empty array but the way the program is built is that I don't necessarily want to exit the loop after dequeue is done, because the user could press a button to restart the process.

I also came across Channel Wires while searching for ways for this problem but since I've never used them before I wasn't sure if I should implement it at this stage. What is the main difference from a Channel Wire to a queue? Is it 'faster'? What would be the disadvantage/dangers of using channel wires instead of queues? Would I have to create two streams from data acquisition to both 'Local' and 'Server' or do it sequentially like the queue?

---

My apologies!  I didn't open your code because I (mistakenly) assumed it was in LabVIEW 2023, which I don't have installed. 

I'm also sorry I didn't explain "Sentinels" well enough.  I notice that you use the "NI" way to close Producer/Consumer Queues -- you "Dispose" the Producer Queue, and depend on detecting an Error on the Consumer's Dequeue to close the Consumer.  What I was suggesting was to have the Producer, when it exits, send one more "Sentinel" packet, such as an Empty Array, and otherwise leave the Queue intact.  The Consumer tests every Dequeue output to see if it is a Sentinel (i.e. tests for "Empty Array?").  For all but the final packet, the answer will be "False" (i.e. "not Empty Array"), so inside the False case, you do whatever you need to with the Array, including Enqueuing it into the second Producer/Consumer pattern.  But when you get the Sentinel's Empty Array (the "True" case), you do nothing except bring a "True" out to the While Loop's "Stop" condition, bring the Queue Reference out, and (now that you already know the Producer has exited) you Dispose the Queue.  [Of course, you follow the same logic with the second Producer/Consumer pattern).

If you look in the Examples that ship with LabVIEW under "Channels", and find Channel Basics, and open Channel Basics Lesson 2, you will see a comparison of a Producer/Consumer "Queue with Sentinel" (which I think represents a welcome change from disposing the Queue on the Producer, causing an Error Exit on the Consumer) with a Stream Channel, where the "Exit" logic is built into the Writer and Reader end-points.  You'll also notice that the "Pipes" carrying the data mirror the (logical) flow of the data, from Producer to Consumer (instead of appearing to "flow" backwards in the Producer, and then branch over to the Consumer).  It's what I call a "better metaphor for data movement/flow".

Bob Schor