Implications of using Named Queues at Scale

[RkTrey]

I have an application that utilizes named queues at a rather large scale--currently around 2000--and I expect the demand for these items to grow as these are essentially created via plant configuration. Without getting into why I'd want to do this, I'm curious to know what are the pitfalls (other than potential namespace collisions) in such a design. 

I've assumed that there's some low level lookup that maps names to references and I'd assume that, if this is the case, that mapping is based on some red-black O(logN) implementation or better. Does anyone know the details regarding this name=>reference mapping or perhaps some information regarding its performance? 

Is there some hard limit to the number of named queues that I can have within an application instance?

[Jacobson-ni]

I know you are limited to 2^20 queues (as well as many other references) so I presume the limit would be the same for named queues.

If I had that many queues I would probably be more concerned about name collisions or just giving data to the wrong queue. I don't know what exactly you are looking to do but if you're at the point where queue lookup is an issue because of how often it is being done I would probably look into more pub-sub or observer type patterns.

[Kyle97330]

I did a quick coding test. 

Creating 5 named queues and then looking up one of those queues at random 1000 times in a row took around 2.5 ms on average. 

Creating 500,000 named queues and then looking up one of those queues at random 1000 times in a row took around 5 ms on average.

(the second one of course took longer to spin up before the timed section to create 500,000 queues)

So while there is a slight slowdown, if increasing by a factor of 100,000 only doubles the lookup time I don't think it's a factor worth considering.

Of course, the memory consumed by those queues if you don't empty them out regularly could be a thing, depending on your use case.