Practical Limits on Number of Queues/Notifiers?

[TurboPhil]

I am toying with the idea of implementing an architecture in LabVIEW that would require the use of many (~thousands) queues or notifiers. I was wondering if anyone has had any experience with working with so many unique queues/notifiers--are there practical limits? Does LabVIEW manage memory effectively at that scale? Are there any caveats that I should be aware of?

A little more background: I'm working on a system in which we receive data from various souces in groups. Each data group consists of several individual channels (basically an array of data). There are thousands of unique channels and hundreds of unique groups. Some channels may exist in more than one group. I would like to implement a Publish-Subscribe system such that a client may subscribe to one or more channels; the Subscription request would publish globally, and each group object would check whether it contains the requested channel. If found, the group object would publish the channel via a notifier, which was created by the client (the reference is included with the subscription request). If multiple groups contain the requested device, then the client will end up with a "get latest" functionality when it queries the notifier (multiple Publishers sharing a single notifier).

[Ben]

I can not speak to thousands but I can speak for hunderds of queues working just fine which calls for a ....

A BIG SHOUT OUT to STEPHEN MERCER, Great job Stephen and your team!

Ben

[SteveChandler]

That is an interesting question. I think I will create some kind of benchmarking code and see where it breaks.

[TurboPhil]

I'd love to see the results of your test, Steve.

That's something I've been meaning to do myself, but I certainly wouldn't mind piggybacking on your tests.

[SteveChandler]

I will post it when (if) I get some time. Right now I am trying to think of how to go about it.

I am thinking of a parent class that contains an array of some other class. Each instance of that class will have two queue references. I will get a timestamp, send that to an object on one of it's queues, that object will get a timestamp and send that on the other queue. Maybe I will include a variant as part of the queue data so I can run it with various data types and sizes.

Just some random thoughts. Does anybody have suggestions on benchmarking large numbers of queues, or, better still, has anyone done this already?

[Ben]

Lack of memory aside, if the queues die I would guess at 65535 queues. But I suspect they will not fail until you get to millions and at that point memory would be involved.

Ben

[SteveChandler]

I only had time to come up with a simple benchmark. But it shows that queues are blazingly fast. Enqueue and dequeue time does not seem to be dependent at all on the size of the data.

I think you are fine if you are talking thousands of queues. Tens of thousands? Probably still fine.

[SteveChandler]

Arrgh! I hit submit too soon, browser crashed and phone rang. Ran out of time to edit..

I was going to ask why the snippet does not work. I created an object based queue tester that does work. It is the results of that test that make me say that you can have thousands of queues.

The snippet above for some reason does not work for more than 100 queues (hangs on dequeue). I know I am missing something obvious.

[Edit:  Ok now the snippet is working fine (after reboot)]

[SteveChandler]

---

@Ben wrote:

Lack of memory aside, if the queues die I would guess at 65535 queues. But I suspect they will not fail until you get to millions and at that point memory would be involved.

Ben

---

That would be a good guess although I am sure it is very educated!

In the snippet above enter some huge crazy number of queues - like a million. In the dequeue for loop typecast the reference to a string and set it to hex display. The queues will be created, the enqueues will all work, then the dequeue for loop will start. It will freeze once the last two bytes of the queue reference rolls over from FFFF.

That's ok - "64K queues ought to be enough for anybody"

[tst]

---

@Steve Chandler wrote:

Enqueue and dequeue time does not seem to be dependent at all on the size of the data.

---

That's because they don't actually touch the data. The reason becomes obvious if you think about it for a second - once you enqueue a piece of data, it can only be dequeued once, so you can guarantee that there won't be any copies of it. That means you can basically just use a pointer to "move the wire" from the enqueue primitive to the dequeue primitive.

Of course, this falls apart if you preview the queue or if you split the wire after the dequeue, but that doesn't affect the dequeue itself.