missing data in queue

[faustina]

I have 4 asynchronous running VIs that share data between each other via queues.  The producer VI puts the data on 2 different queues; one for a consumer VI to write data to a file and one for 2 consumer VIs to display data.  The first display VI takes all the data (in waveform format) and converts it to a string for display in a table.  The second display VI takes a subset of the data and converts it to DBL for display on individual indicators.

I am not experiencing any lost data being written to the data file.  I am experiencing missing data for the displays.  I put a test sine wave on one of the data channels to figure out which queue output is having the problem.  There are about 90 channels of data being collected.

The table VI only displays the latest data point for each channel in the table.  I didn't notice any missing data until I wired the test channel into a chart.  I thought could be an update issue.  However, ...

The subset VI displays all the data from the test channel in a chart and there is missing data.  Out of all the channels only one is being displayed.

The producer VI puts the data on the queue using Lossy Enqueue Element function.  The consumers get the data from the queue using the Preview Queue Element function.

Any ideas on where to begin to diagnosis the problem?  I would post code but I haven't successfully posted any files to this forum.  Is there any other location I could post?

[johnsold]

You should only have one place where data is removed from a queue. If you need the same data in two or more places, you need additional queues or some other method of moving the data. User events and functional gloable variable VIs are possibilities. 

Some browsers seem to have trouble posting VIs. Try putting the VI in a zip file.

Lynn

[chembo]

So, you never dequeue elements from the queues, just a preview. Are you using queues with just 1 element?

[crossrulz]

You really should be using a User Event to send your data.  In fact, I would use a User Event for your data logging also.  Then you just have to send one message (no having to handle multiple queues) and then everybody who is registered for that event will get it.  No loss of data, very efficient, does not use a hack to get the data there.

[crossrulz]

---

faustina wrote:

The producer VI puts the data on the queue using Lossy Enqueue Element function.  The consumers get the data from the queue using the Preview Queue Element function.

---

So you never get elements off of the queue and you use a Lossy Enqueue...So once the queue fills up, you will never get any new data into the queue...That sounds like a problem.

[chembo]

---

@faustina wrote:
.........................

The producer VI puts the data on the queue using Lossy Enqueue Element function.  The consumers get the data from the queue using the Preview Queue Element function.

..........................

---

I guess this is probably a queue with limited size, for example single element queue, or whatever size, which keeps just the last X values. Otherwise, if the size is unlimited, the Preview will return always the same top element of the queue.

I use sometimes this single element queue with lossy enqueue/preview in order to keep the latest value of a continuous measurement, when data loss is not relevant.

[Bob_Schor]

To me, Fixed Length Queues do not imply the use of Lossy Enqueue.  I have a Queue that I use as a buffer when streaming data from a Network Stream to disk.  I'm getting data packets 20 times a second, and I know that I can easily write the data many times faster.  I create a fixed length queue of maybe 10 packets and pre-allocate it -- I could use an unlimited length Queue, but then I have no control when the Queue manager decides to allocate more memory to my Queue in the middle of an experiment.  For such a Queue, I use a "normal Enqueue" process, as I'd rather Block and get an error or a temporary backup of my Network Stream than lose data without knowing about it.  Of course, I do some benchmarking to be sure I have a sufficient "safety factor" on my fixed length Queue size.

I also use Fixed Length Queues as "finite buffers", or "remember the last N elements".  Here I do use Lossy Enqueue.  One use I made of this was to save Video buffers to allow me to view images starting 5 seconds before I registered the triggering event -- I just stored the "PreEvent" frames in a Pre-Event Lossy Queue.

Going back to the Queue from the Network Streams, I also want to display these data.  I create another Display Queue which I fill as I am dequeuing elements from the Network Streams.  If I had two displays that needed these data, I'd build two Queues.  Again, for reasons of economy and efficiency I might make these Fixed Length Queues (maybe 5 times the maximum recorded Queue length tested with unlimited-length Queues -- memory is cheap), but would not fill them with Lossy Enqueue functions.

Bob Schor

[crossrulz]

---

Bob_Schor wrote:

Going back to the Queue from the Network Streams, I also want to display these data.  I create another Display Queue which I fill as I am dequeuing elements from the Network Streams.  If I had two displays that needed these data, I'd build two Queues.

---

To me, that is a prime spot for  User Events.  You only have to worry abount 1 message and whoever wants it can register for it.

[Bob_Schor]

Crossrulz,

     I'm interested in your suggestion for using User Events when you need to send queued data to two locations.  I'd not been aware that you could do this, and am not sure I know how to implement it.  I did go back and read the (2014) Help for User Events, and found (under Caveats and Recommendations) the following:

You can register for the same user event multiple times by using separate Register For Event functions. In this situation, each queue associated with an event registration refnum receives a copy of the user event and associated event data each time the Generate User Event function executes.

However, I'm uncertain how to do this in such a way as to have my data be processed by two parallel processes (and would happen if I used two Queues).

     Can you point to an example, or provide one?  This sounds like an Interesting Technique to Learn ...

Bob Schor

P.S. -- I am familiar with User Events and have used them numerous times.

[crossrulz]

---

Bob_Schor wrote:

     Can you point to an example, or provide one?  This sounds like an Interesting Technique to Learn ...

---

I do this all the time.  My prime example is with data that needs logged and updated to the GUI.  So I have one loop that processes data and then sends it by generating a User Event.  I then have a data logging loop that is registered for the event that can log the data.  I will then have another loop that handles the GUI that is registered for the event to just update an indicator.

This example is a lot dumber than that (it just updates two indicators).  Saved in 2014.  Let me know if you need an older version.