Engine-to-Engine (E2E) Tag Exchange Module Documentation

[Knoxberry]

Purpose

The purpose of this thread is to provide provide a portal on the NI community forums for discussions of the DCAF E2E Tag Exchange Module. This thread will also provide a general overview of the module. The most up-to-date documentation is hosted on GitHub. See https://github.com/LabVIEW-DCAF/E2E for more details.

Overview

The Engine-to-Engine Tag Exchange (E2E) module is meant to simplify the exchange of tags between
separate DCAF engines. Rather than configuring output and input channels manually using a specific
I/O Module to exchange tags, the E2E module handles channel configuration automatically when a
tag is selected to be shared between engines. The editor node of the E2E module also allows the
user to determine the transfer mechanism for tag exchange. The transfer mechanism determines
the allowable data types, the pairing depth (inner-target, inter-target, both), and other specific
configuration information. The module was designed to allow for users to create and add new
transfer mechanisms. The first two implemented mechanisms were UDP and Single Element RT
FIFOs.

Required Software
• LabVIEW 2015 or Later
• DCAF Tag Editor 2.1.3.172 or Later
• DCAF Standard Engine 2.4.0.8 or Later
• DCAF UI Reference Module 2.3.0.12 or Later (For Example)
The required DCAF components can be downloaded and installed from the LabVIEW Tools Network
Repository through VI Package Manager.

Using the Module in the Standard Configuration Editor

1. Adding the Module to a DCAF Configuration

The VI Package Manger installer for the E2E Module will attempt to add the E2E Module's

destination directory to the DCAF Configuration Editor's plugin search paths. If successful, you will be
able to add the E2E Module to your DCAF Engine's configuration by right clicking the engine
configuration and selecting Add>>Other>>E2E Tag Exchange.
 

If not, you will need to add the E2E Module's destination directory to the Plugin search paths manually. The default installation directory for the E2E module is \<LabVIEW>\vi.lib\Data Ahead AG\DCAF\E2E Tag Exchange Module. Add the directory to the plugin search paths by selection Tools>>Edit Plugin Search Paths... from the toolbar of the DCAF Standard Configuration editor.

2. Selecting a Runtime Implementation

Once you have added the E2E Module to a DCAF Engine, you can select its runtime implementation by using the drop-down box on the Node Pairing tab of the configuration window.

The runtime implementation determines which runtime component is used to exchange the tags, which pairing distances are acceptable, the data types that can be exchanged, and other specific configuration information.

3. Pairing the Module

E2E Modules exchange tags by pairing with each other on a One-to-One basis. When an E2E module is selected, the “E2E Nodes” table will display all other E2E modules in the configuration.

The information displayed for each module is:

1. Display Name – The module’s current alias
2. UID – The unique identifier of the module
3. Runtime Implementation
4. Distance – The relationship of the selected E2E module to the module in the table
   1. 0 – The module is under the same engine
   2. 1 – The module is under a different engine in the same target
   3. 2 – The module is under a different engine in a different target

The row for a module will be disabled given one of the following conditions:

1. The runtime implementation is different
2. The module is already paired
3. The runtime implementation doesn’t support the pairing distance (ex: RT FIFOs can’t exchange tags between separate targets)

Pair a module by selecting an enabled row in the table and clicking “Pair”. Use the “Unpair” button to unpair the paired modules.

4. Exchanging Tags

The “Tag Exchange” tab displays the following information:

1. Available Tag List – The tags that are currently available for exchange
2. To External Engine – Tags currently configured for transfer to the external engine
3. From External Engine – Tags coming from the external engine
4. Filter – A string input used to filter results with a match pattern function
5. Prefix – A string concatenated to the front of all input tag names
6. Suffix – A string concatenated to the back of all input tag names

Tags are considered available when the data type is supported by the runtime configuration and the tag isn’t already being transferred either to or from the current module. To transfer a tag, select the tag from the available tag list and click the “Move one right” button. Transfer all tags by clicking the “Move all right” button.

Remove tags from the list by selecting a tag from the “To External Engine” List and using the “Move one left” and “Move all left” buttons.

The tags will now appear in the “From External Engine” list in the paired module.

Renaming tags corresponding to each input/output channel is not supported because the E2E module uses each tag’s name to sort tags for transfer. Use the “Prefix” and “Suffix” inputs to manage tags coming from an external engine.

5. Editing Transfer Mechanism Specific Information

If a transfer mechanism has specific configuration information, a new tab will be available.

UDP Configuration

The UDP runtime implementation has the following specific configuration information:

1. Output Configuration
   1. Address – The network address to write to
   2. Remote Port (write to) - The UDP port to write to
   3. Local Port (write from) – The UDP port to write from
2. Input Configuration
   1. Address – Always localhost
   2. Local Port (read from) – The UDP port to read from
   3. Last Update Tag? – If selected a channel of type double will be created. Use the corresponding drop-down selector to map a tag to the channel. The channel contains a timestamp with the last successful read time. The generate button will create and map a tag to the channel
   4. Estimated Total Array/String Size (bytes) – This input will be available if the tags from the external engine contain data types of dynamic sizes (Strings and Arrays). The UDP runtime component will attempt to read a packet from the input port with a max size determined by the size of the statically sized tags, the packet header, and the value contained in this input plus a small buffer
3. Estimated Output Packet Size (bytes) – The combined size, in bytes, of all output tags with statically sized data types plus the output header
4. Estimated Input Packet Size (bytes) – The combined size, in bytes, of all input tags with statically sized data types, plus the input header, plus the value in the Estimated Total Array/String Size (bytes) field

Licensing

Copyright 2018 Data Ahead AG

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this plugin except in compliance with the License.

You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

[NickNZ]

With DCAF, I am currently sharing data between a cRIO and PC via the UDP module, and between multiple engines on the cRIO via CVT. 

I gather could use this module as an alternative. What are the pro's and cons of both approaches?

I have quite a large number of tags. 

[Jacobson-ni]

For the case of sharing data between cRIO and PC, there isn't much of a functional difference between the E2E and UDP module and the UI's are pretty similar. Personally, because the UDP portions of the module are so similar, I've been leaning towards using the E2E module because I think the code is generally cleaner and the points of extension are clearer and better documented (not that it matters too much as UDP is already implemented).

Comparing CVT to E2E is probably a bit more difficult. First, their underlying transfer mechanisms are different. CVT is essentially getting/setting array indeces in an FGV while E2E uses RT FIFOs. I don't know which method would be more efficient but I would suspect both are far from inefficient.

On the UI side, the E2E module handles a lot of the tag pairing for you. If you write a tag on one of the paired modules, the other engine will map the tag of the same name accordingly and will even create a new tag if one doesn't already exist. When using CVT you are going to have to set one module to write a set of tags and then manually configure the CVT module on the other engine to read tags with the same name and data type and the editor will not show any errors if you haven't done this correctly.

Now the advantage of using the CVT module is that you can set up a 1:N communication scheme while E2E is strictly 1:1. Additionally, you can use the CVT module to easily communicate to non-DCAF portions of your code.

[NickNZ]

Thanks for the great comparison:

I am 1:1.

I have had errors with tags not mapped correctly in CVT. So its good feature that this module is better at sorting them out.  

[NickNZ]

Hi 

Another question,

If the E2E uses RT FIFO's I assume that means the writer and reader have to be running at the same speed?

With CVT I have a higher speed acquisition and logging loop (200Hz), which passes its values to the CVT multiple times. 

I have a slower engine (5Hz) that read the CVT and sends it across the the UI via UDP module as there is no point going faster for the UI.

I am also have slower loops (1Hz) that get data to CVT which is read multiple times and sent via UDP.

Anyway going to try it out as need to rebuild project by module to figure out why my cRIO is maxing out a couple of cores.

It may be that using both CVT and UPD modules (faster to slower loops) and E2E solutions (slower engines to UI) is the best solution.

Cheers

Nick

[Knoxberry]

Hi Nick,

The RT FIFO implementation of the E2E Runtime uses a lossy (timeout 0, overwrite on timeout) single element FIFO for tag transfer. This meets the DCAF paradigm of tag based acquisition and processing, so there is no problem with having engines running at different rates. The input module will receive the latest value from the output module (if there is no element in the FIFO when the input module reads, the value isn't updated in the tag bus). 

In my opinion, the CVT module is great for transferring DCAF tags to non-DCAF portions of a complete application, but it can be cumbersome to manually map tags between two CVT modules running in separate DCAF engines. If your tag transfer is completely internal to DCAF, I'd recommend the E2E module as it can aid with the mapping. Matt gave a great overview above.

If you have feedback for the E2E module I'd love to hear it.

Cheers,

[NickNZ]

Hi Ian

OK that makes sense. I tired it out yesterday as had a bug that was using a lot of CPU on my cRIO. 

So I rebuilt by configuration one module at a time to find out the dodgy module (It turned out to be one of mine). So used your E2E UDP version instead of  CVT/UDP. 

With only a couple of modules there didn't appear to be any performance difference (Not that I was really expecting to see one on a cRIO-9049). However I do like the tag mapping.

Initial comments:

1) Generate includes only includes the FIFO E2E module, not the UDP one. 

2) I had two E2E modules going to the test PC host. I don't know much about port numbers and of course got a clash. So I just incremented the port numbers on the 2nd Module and it worked. No mention about that in documentation. 

At this stage I like the E2E module and I do feel it will lead to a cleaner implementation. Still need to use it a bit more if you want more feedback. Thanks for your contribution. 

[NickNZ]

Hi

I am using E2E to pass data from a cRIO to PC via UDP.

I had my PC receiver UI engine running at 5Hz (Since you can't see faster anyway)

I had the cRIO sender engine running at 10Hz. 

It appears the that the UDP packets are being buffered. I can run for a while and then when I unplug the network cable the values continue to update as the buffer is consumed.

This is contrary to my expectations, I expected every 2nd UDP packet to be lost / overwritten rather than be buffered. 

Not sure if the UDP module behaves like this... as I haven't invested it yet. 

Regards

Nick

[Knoxberry]

Hi Nick,

Thanks for the feedback. 

1) The scripting tool for the Run-time dependencies should only add the Runtime object of the implementation you're using. Does it fail to add the UDP implementation when you are using UDP? If so, it's a bug and I'll add it to the issue tracker in GitHub.

2) I'll add a request to at least add a link to the UDP documentation in the E2E Module Documentation. It could definitely be more clear about the port configuration.

As for the backlog of packets you're seeing, how are you passing the data from the DCAF Engine to the UI on the PC? Maybe that's where the buffering is happening. From my understanding of the LabVIEW UDP implementation, packets aren't buffered. Any packets at the port will be overwritten by new packets arriving if there isn't an active listener. Maybe someone else from NI can chime in to verify this. One way you could test this is to see if the delay exists before disconnecting the network cable. 

The runtime implementations of the E2E module and the UDP module are almost exactly the same. I mostly just cleaned the code up. My initial idea was to use existing modules as the transfer mechanisms for the E2E module. For example, someone could take the runtime of the CVT module and modify it to match the paradigm of the E2E module. The main portions of the configuration and editor classes would be the same regardless of the transfer mechanism (runtime implemenation). 

Cheers,

[NickNZ]

Hi Ian

apologies for the delay in getting back.

1) I haven't verified this yet. 

2) It just bit me so I had to take a look.

Setup:

cRIO1 = MODBUS Ethernet master.(Simulated with a UI for testing and debug purposes)

cRIO2 =DCAF host / engine / modules

              1 Hz Engine

                    MODBUS Module Ethernet (Slave)

                    E2E UDP

PC = DCAF host / engine /modules 

          0.1 Hz Engine ( - this was my bug as supposed to also be 1Hz, had an extra 0 in dt field)

                     E2E UDP

                     UI Module to show MODBUS registers

(I have other modules etc but above is relevant to what I am discussing)

My bug was I thought the UI engine was running at 1Hz, but I accidentally put 10000 into dt for the UI engine, so it was running once every 10 seconds. So when I ran the UI, it currently takes about 5 seconds to sort itself out, during this time Windows is buffering UDP data. I then saw 1 UI update. But 10 seconds later when the UI was updated again, there were now 15 packets in the Windows UDP buffer, and E2E picked up the oldest. So it looked like it wasn't working by the time I started modifying the controls on my MODBUS master. (As I didn't run the UI long enough to see the 2nd update, or if I did it still didn't match what I was sending to the MODBUS). I could see with WireShark the packets coming in at 1Hz - so I knew it was the UI end. 

Your optional last update tag was useful in finding that the UI loop was running slower than I expected.

I fixed my UI loop to 1Hz and it "works" but still with ~5 second lag and write and read rates are roughly equal.

So I had a dig into your E2E code and compared to the UDP code. 

I agree your E2E implementation and coding quality is much better than the UDP module. 

However the UDP module does read from UDP until error 56, (effectively flushing the buffer to get the newest packet from the OS's UDP buffer), whereas you added the comment in read and adjust input size.vi

If a data packet is available,
read will return with out error.
If not - error 56. Flushing isn't
necessary

I disagree with this statement.

This article discusses the buffer on host. (Outside of LabVIEW's control)...

https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z0000019YLPSA2&l=en-NZ

For DCAF we only want the latest current values, so we do need to flush the buffer. 

I will see about creating a pull request for some updated code. 

I tested by setting a control to increment every second on the master MODBUS. The new receive code with the UI run at 1Hz it incremented every second with about a 3 s delay (delay is reasonable & expected). I then slowed the receive UI running to 0.2Hz the and saw that the UI updated every 5 seconds and showed the latest count (so 5,10,15,20 etc) again with about a 3 second delay.

I then restored your original read and adjust input size.vi and proved that Windows was buffering the data. The UI incremented by one count every 5 seconds and got further and further behind the counter on the MODBUS master which was running 5x faster. 

3) Found another issue, when you have a lot of tags you can't scroll from external list. See attached picture.