Global Queues Palette

[‎‎‎‎‎‎]

Proposed:

• New "Global Queues" palette with with both "queue name" and "application scope" input required, on Global Obtain Queue and Global Release Queue.
• Global Queues to be accessible to ALL projects / applications running on the same host, but only if referring to (the same) queues with the same queue name AND application scope (not just queue name, but both, to allow for scope partitioning).
• New "release mode" input to Global Release Queue (enum, 3 options): "release" (single local ref, as per normal queues), "force-release" (all local refs, as normal queues with forcing TRUE), and "global force-release" (all refs with the same name AND application scope pair, globally).
• Due to multiple projects / applications referencing the same queue locally, a few further VIs or methods would also be productive, such as: get total number of local references to queue, get total number of global references to queue, singularise all local references to queue (release all but the latest 4B ref), these are mainly related to monitor & control the accumulation of 4B queue refs in a direct closed loop fashion (as opposed to merely keeping track of them in an action engine).
• Due to further increased complexity, a queue attribute / property (and a Global Queue Status VI output revealing this) specifying the enqueuer application -> library -> VI chain for each enqueued / dequeued / flushed item, could be added. This could also be used for security & access scope purposes if desired (i.e. custom code to reject the dequeueing of any queue element not enqueued by a specific enqueuer, etc.).
• For monitoring & debugging purposes, a VI to output names of all queue names associated with a particular application scope would also be useful. Application scope name would still need to be known, but code for dynamic allocation of queue names within the same application scope could be designed with much better compliance.
• Although this can be considered a queue functionality extension, having a separate palette (e.g. same queue icons, different colour) is subjectively better, makes it easier to read code visually, less error-prone, and more consistent to implement with the crucial new Global Release Queue "release modes" proposed.
• Essentially the extension of named queues to be accessible by multiple distinct projects / built applications (assumed to be long-awaited by many), but with being able to specify the pool of interacting applications explicitly, hence "application scope".

Purpose:

• Direct queue use across multiple projects / applications on the same host. Similar comms can currently be implemented by using network streams (acceptable lossless FIFO), but these are grossly inefficient and slow compared to queues in memory, require coded polling to check stream items available to read, and are exposed to, as well as dependent on networking protocols and even firewall configuration. There should be no need at all to use any networking for separately built daemons running on the same host, and no need to do (coded) polling in particular.
• This would also allow for a new approach to plugin architectures where plugins can be implemented as fully decoupled, compiled, standalone daemons each (a potential for further explicit optimisation for parallel processing). Process call overhead and memory implications could be managed by means of coding directly (daemons could be individually configured to be either launched on demand, or launched automatically to await instructions / data on standby). Develop new plugin adhering to engine specs on queue datatypes only, build it into an exe using any version of LabVIEW, set single or multiple instance mode in ini file directly, supply to client, done...

[Intaris]

Won't Windows complain about one process (application) trying to access the memory space of another?

[‎‎‎‎‎‎]

I think not, if done properly, but global queues could be managed by the LabVIEW environment / run-time memory space, with the accessor applications validated by queue name and application context variables simply requesting their own actions from these, to avoid access violation errors. Almost like a memory server with clients, which could simply be launched if any LabVIEW project / built application uses global queues.

The core idea, albeit typically appearing in a simple unmanaged "why aren't my queues working in another exe" context, has been around for about a decade now, from conclusion of other LabVIEW discussions. It is also worth noting that computer forensics tools such as WinHex, do direct memory access of other processes already, but of course this isn't necessarily required for the proposed feature - and probably wouldn't be the best approach to it either.

[wiebe@CARYA]

I'd say the RTE of two LV applications are two instances of the RTE. There's no global RTE that runs executables. I could be wrong.

There are techniques that allow (are designed for) shared resources, like memory mapped files, named pipes, Microsoft Message Queuing and application queues. I've only used the first two. It won't be easy to make an (efficient) global queue using them. Last two seem to be designed for this. Windows only though... NXG could probably use RabbitMQ, as that's already used.

Technically, there are solutions...

[‎‎‎‎‎‎]

Sure, but I strongly suspect both end-users and architects would rather deal with this as LabVIEW code. I'd think most end-users, should they need to resort to workarounds as per current situation, would much prefer to get the LabVIEW architect to implement this as network streams, even if there's some modest coding overhead with network streams being single endpoint, single direction only. One can easily create a stream for each purpose & direction, though, as it would have to be the case with named pipes if I am not mistaken (I might). However, even if a mere workaround, doing this in LabVIEW allows for a single source of delivery of a particular implementation. Admittedly, the network stream one isn't fast or efficient, but consistent, lossless, FIFO, fairly traceable, and most importantly, not an external piece of software to rely on. To my eyes, any external software requirement for applications coded and compiled from a programming language per se, only reveals limitations. I feel I should be able to do this relying on LabVIEW alone, and do this properly (as opposed to workarounds). Hence the suggestion submitted here as "idea".

Ah OK, under "...global queues could be managed by the LabVIEW environment / run-time memory space..." I didn't mean to imply a single RTE that just runs all executables and also serves as a global queue handler itself, but rather all RTEs being capable of managing (controlling) a global queue server, i.e. installed by RTE, but run as a separate service, on Windows. This I believe should also be much easier to implement (not that ease on itself typically has any value on its own though). Software running their own service(s) with controlled access privileges, is a very common occurrence.

Memory mapped files sound great, I'm just not sure about the efficacy of this, considering it's a virtual memory segment (a physical data block / file). What's the situation there with encapsulation, trivially encapsulated by r/w access flags, or not so simple? (Not sure.) Perhaps the speed issue diminishes with ramdrives soon anyway...

[Dmitry]

I would say this Idea boils down to a limited, LabVIEW-only flavor of middleware. This might be fine for a light-weight communication solution between NI Targets, but at some point LabVIEW application might need communicating with non-LabVIEW applications (man, do I hate STM library). So why limit your scalability from the beginning?

There is a LabVIEW DDS wrapper from RTI known as DDS Toolkit for LabVIEW. It comes standard with LV2017 (located in Data Communication palette) and provides substantially more communication options than global named queues. Moreover, it is an industry standard - allowing communicating with programs written in other languages. A quote from above link:

This toolkit is built on RTI Connext^® DDS software, the leading connectivity framework for Industrial IoT applications. Applications developed using the RTI DDS Toolkit can seamlessly communicate with C, C++, C#/.NET, Java and Ada applications developed using Connext DDS. In addition, because Connext DDS complies with the Data Distribution Service (DDS) standard, LabVIEW applications based on this toolkit can interoperate with applications developed using any DDS implementation that complies with the DDS-RTPS wire interoperability protocol.

So why use scarce LV R&D resources for implementing a flimsy squeaky wheel ?

[‎‎‎‎‎‎]

RTI DDS seems great (haven't seen this yet), especially for true distributed systems.
There are a few reasons to still want queues, however:

• RTI DDS seems to require a significant architectural / run-time overhead for the brutally simple functionality proposed
• LabVIEW should be able to do this without requiring a paid (?) add-on, at least on the same host PC, natively, or is this the case with LabVIEW 2017 now? (haven't installed that alliance distro yet)
• Consider ratio of use-cases: true distributed applications versus inter-process comms requirements on the same host
• Scarce LV R&D resources, even if true, aren't my problem, but lack of such a basic feature leads to less sales (and one less powerful argument to convince clients of moving to LabVIEW)
• I'd be moderately curious to compare sheer efficiency & efficacy (speed, mainly)
• As this is a LabVIEW idea exchange, I trust you see the scope for a "LabVIEW-only" flavour, too, at least on the basic level of enabling a queue type to do comms across applications, at least on the same host (without the overhead of reader-writer subscriptions and corresponding, directionally defined tunnels, ideally)

[AristosQueue (NI)]

Queues are, by design, single-process communication tools. This is a key part of their performance. Inter-process communication, even on the same machine, requires entirely different setup, synchronization, and error behaviors. This proposed idea is predicated on the idea that this is "the brutally simple functionality," but it is actually very complicated. The requirement to handle more complex messaging scenarios is why other communication tools in LabVIEW exist. You will need to use one of them.

We will not be implementing this idea.

[Darren]

 National Instruments will not be implementing this idea. See the idea discussion thread for more information.