queue not queueing

[grahamwebb]

I have a producer / consumer architecture - data are being read (from a TCP read) in the producer loop and placed on a queue. The consumer is reading the queue and doing lots of modelling, analysis and display. This architecture was chosen because the data doesn't arrive at a consistent rate from the server - due to network load etc. I am reading 'number of elements on queue' to get a rough idea if there is latency in the system. 

The server sends a frame number, which I save using the TDMS format. I recently noticed that when the number of elements on queue increases, the frame number skips, i.e, when data are buffered on the queue, they don't seem to appear when the elements are dequeued.

What's going on?!

Unfortunately I can't post the code, but I will try and produce a cut down version. 

[DFGray]

It sounds like you have a race condition somewhere, but without code, it would be difficult to say.  You may also be having issues with out-of-order execution, since the timing of TCP is not guaranteed.

Please post what code you can so we can give better feedback.

[Ben]

---

@DFGray wrote:

It sounds like you have a race condition somewhere, but without code, it would be difficult to say.  You may also be having issues with out-of-order execution, since the timing of TCP is not guaranteed.

Please post what code you can so we can give better feedback.

---

I would suspect a "lossy" queue myself.

Re: TCP

The order is guarenteed.

Ben

[grahamwebb]

In the process of cutting down the code to make an example, the problem went away - all I've done is replace the TCP read with a random number generator + delay. Will post something later..

Thanks

[Ben]

---

@grahamwebb wrote:

In the process of cutting down the code to make an example, the problem went away - all I've done is replace the TCP read with a random number generator + delay. Will post something later..

Thanks

---

now THAT sounds more like a race condition.

Ben

[dan_u]

What's your queue size? Do you use a timeout when enqueueing elements?

The queue size should be big enough to keep a few elements that are received over TCP, but of course you have to make sure that your consumer loop can process the data fast enough so the queue does not "overflow".

[grahamwebb]

The queue size is default (unlimited).

The queue builds up when the PCs processing load goes up - when an email arrives etc. You can see this by reducing the delay time right down to 1ms and click "real-time operation" then do something else with the PC - shake the vi window around the desktop for example. The queue should build up, unless you've got a good PC!

The problem I have with my real program is that data being queued are lost.

Other difference with my real program - data are coming from a TCP read function, there is no delay in the producer, the data are text not numerical, and the consumer is huge. I'm not sure how to set a sensible timeout. There's also no handshaking with the server.  

G

[grahamwebb]

doh!

Here it is  ..

[Taki1999]

What's with the pre-allocation and intialisation part?  It looks like you're just stuffing and flushing the queue 999999 times.  A single flush queue should be sufficient.

[Taki1999]

Also, I didn't see any data loss while running your example.

You probably want to create a mechanism that stops your consumer loop when your producer loops stops though.