time consuming FindRoute function

[MvB_PI]

We are using 3rd party switching and we have a configuration of four 75x4 matrices with additional isolation relays.
All 4 matrices are interconnected to each other.
In NISE (3.0) this configuartion ends up with about 300 hardwire connections.

The FindRoute call returns the correct route and a connect route establishs the correct switch path.

Unfortunately a single FindRoute call takes about 15 seconds for the calculation.

Is there a way to decrease the time for route finding?

Thanks in advance

Matthias

[chericks1]

Hi Matthias,

I'm sorry to hear that you're having trouble.  I'll work with you to find the reason for the long FindRoute execution time. 

So far, I've been unable to reproduce the 15 second execution time for the FindRoute call.  I attempted to reproduce this with 4 NI PXI-2532 matrices, each one in its 2-wire 4x64 matrix topology.  All 4 modules were simulated in MAX and configured in NI Switch Executive to have a Bus that connects all the columns to one another as shown below.

All rows were configured to be Reserved for Routing.  The configuration is typical and allows for column-to-column connections - up to 16 at a time (by using a separate row each time for an interconnection).

Is this similar to your intended use case? 

Using the above configuration, I created a VI to test the FindRoute call execution time.  As shown in the picture below, the execution time for a column-to-column route that spans multiple NI PXI-2532 switch modules was on the order of 20 ms.  Experimenting with different routes, the highest time duration result I was able to witness was on the order of hundreds of milliseconds.  

Background on FindRoute:

The NI Switch Executive FindRoute call leverages IVI class driver calls, which then calls into an IVI specific driver.  For NI switch modules, this IVI specific driver is NI-SWITCH.  For 3rd party switch modules, there is a 3rd party IVI specific driver.  The execution time difference between NI-SWITCH and the 3rd party IVI-compliant switch driver calls may be the reason that I'm not able to reproduce the issue. 

Questions:

1. Which 3rd party switch driver / hardware are you using? 

2. Are you able to reproduce the issue when simulating the 3rd party switch hardware?

3. If so, then can you post a) an exported .xml file of your NI Switch Executive Virtual Device and b) a copy of the IVI Configuration Store .xml file that is on your machine? 

Procedure for (a), exporting Virtual Device file: http://zone.ni.com/reference/en-XX/help/370404F-01/nise/maxexport/

Procedure for (b), copying IVI Configuration Store file: http://zone.ni.com/reference/en-XX/help/370404F-01/nise/sysdeploy_export/ 

As long as we can simulate the hardware and still reproduce the issue, then I should be able to install the 3rd party driver, use your IVI Configuration Store .xml file, and import your NI Switch Executive Virtual Device to successfully reproduce the 15 second FindRoute call duration.

Let me know if you have any questions!

Chad Erickson

Switch Product Support Engineer

NI - USA

[MvB_PI]

Hi Chad,

thanks a lot for your response.

The described behaviour we do see in simulation mode as well. It is not the hardware which causes the delay it is the finding route function.

For a better understanding of our customer's matrix configuration a sketch is attached (zipped).

There are 2 Pickering LXI devices SX1 and SX2, each of them has 6 Sub Units, 3 Matrices 75x4 and 3 units with 75 SPST relays.
These units are interconnected in a kind of quad configuration.

The hardwires in SE are described as hwa1...75, hwb1...75, hwc1...75, hwd1..75, hwe1..4 and hwf..1..4

The UUT side alias are A1...75 and C1..75, the Tester side alias are B1...75 and D1...75.

A FindRoute from
A1 to B1 takes about 15s
A1 to D1 about 1.5s
C1 to B1 about 1.5s
C1 to D1 about 15s
A1 to C1 about 2s
B1 to D1 about 2s

The configuration sketched is only a part of a much bigger system.

Please see also the attached the virtual device xml and the IVI config store (zipped).

Thanks.

Best Regards
Matthias

[MvB_PI]

Hi Chad,

please see attached the 3rd party IVI driver used within the previous sent configuration.

Regards,

Matthias

[chericks1]

Hi Matthias,

Thank you for the detailed information.  I was able to reproduce the behavior. 

Before today, I've never seen such long FindRoute execution times.  After investigation, I believe that I know why we're experiencing this issue with the Pickering LXI devices and why I have not experienced a similar issue with NI devices.

Using NI Spy, I was able to monitor the number of CanConnect calls that the Pickering IVI specific driver was executing.  You can do this too.  NI Spy is installed with the IVI Compliance Package:

Start >> All Programs >> National Instruments >> NI Spy, or

"C:\Program Files\National Instruments\NI Spy\NI Spy.exe"

When monitoring with NI Spy, a FindRoute from A1 to B1 required 27 seconds to execute.  There were over 500,000 IVI specific driver calls, and the majority of them were CanConnect.

Curious if a similar setup using NI hardware would require the same amount of time to execute the FindRoute, I created an 8x200 matrix (with SPST isolation) utilizing 4 NI PXI-2569 SPST switch modules (each configured as a 100-SPST) and 4 NI PXI-2532 matrices (each configured as a 1-wire 4x128 matrix), as shown below.

The A1 to B1 route only required 385 ms and less than 3000 IVI specific driver calls (with only 200 or so CanConnect calls)!

So, what was the difference? -> The number of possible interconnections per module.

The number of channels on one Pickering switch module (comprised of 6 Sub Units, 3 75x4 Matrix units and 3 units with 75 SPST relays) far exceeds the number of channels on any one NI switch module.  When there is no direct route from A1 to B1, FindRoute uses hardwire information and CanConnect to attempt channel-to-channel connections within each switch module until an optimal route path is determined. 

With the NI configuration, FindRoute was able to quickly propagate through each of the possible connections within each NI switch module until it determined an optimal route from A1 to B1. 

FindRoute was not able to do the same with the Pickering switch module; optimized to perform on one module at a time, and not on Sub Units, FindRoute must check for possible connections throughout the entire Pickering module. 

For comparison, refer to the NI configuration above.  When finding a route path from A1 to B1, FindRoute notes the hardwire and begins searching for a path on the PXI-2569 switch module.  CanConnect is called to attempt ch0->com0 and the call returns that this path can be made.  The ensuing hardwire now allows FindRoute to search for a path on the second PXI-2569 switch module.  Once again, CanConnect is called to attempt a ch0->com0 connection.  The result is Path Available, and the B1 destination is reached.  Other paths are then attempted, but this initial path is quickly determined optimal.

When finding a route path from A1 to B1 on the Pickering module, FindRoute attempts to directly connect A1 to B1 (since both connections are now on the "same" switch module).  CanConnect returns Path Unsupported, so FindRoute notes the hardwire and begins searching for a path.  The key difference here is that it must now search for a path between the current channel and every other point in the entire module, even in different Sub Units.  This is the reason for the 500,000+ CanConnect calls. 

Is there a way to decrease the time for route finding?

1. Instead of A1 to B1, Search for B1 to A1

As FindRoute determines the optimal path from a Startpoint to an Endpoint, and the configuration above is not the same from left-to-right, as it is from right-to-left, you can attempt reordering the FindRoute inputs.  This may not always help you, as in the end the duration of FindRoute depends on the number of possible route paths between the two channels.

2. Matrix Reduction

Matrices increase the number of possible route paths because of column-to-column and row-to-row possibilities.  If you can eliminate a matrix in the configuration, then you can speed the execution of FindRoute.  In this particular configuration, is it possible to expand the matrix from top-to-bottom instead of from left-to-right?  In other words, create a 4x300 matrix instead of an 8x150?  I know you stated that the example configuration was part of a larger one.  If possible, then I believe this would eliminate a number of possible route paths, and all connections would be column-to-column.

3. Reserved for Routing Reduction

Reserved for Routing paths create route possibilities.  Eliminate Reserved for Routing paths to speed the execution of FindRoute.

4. Modular Architecture

FindRoute is optimized for routing across multiple switch modules.  Use a modular architecture instead of a multiple-bank or Sub-Unit solution for a faster FindRoute execution time.

Let me know if you have questions or comments!

Chad Erickson
Switch Product Support Engineer

NI - USA

Message Edited by Chad PSE on 11-19-2009 09:36 PM

[MvB_PI]

Hi Chad,

thanks again for your extensive explanation which gave us the idea what to change.

Due to the flexibility of the Pickering IVI driver the former model of the configuration had been easily changed from
a Matrix+Isolationrelays to a Matrix with Auto-Isolationrelays.

Means looking at your latest diagram the SPST relays inbetween the matrices (your PXI-2569) and on the right side (your other PXI 2569)
had been omitted. Physically the Isolation relays are there and switched but no more seen from the IVI driver.

As before the 4 matrices are interconnected in X (col) and Y (row) direction and these interconnections are defined as hardwires in SE.
All Y lines are reserved for routing. Finding a Route from left to right or right to left takes now less than 50ms.

There is a need to loop back on the left side up to 8 signals at the time. After 4 loopbacks and all Y buses on the left side have been used the
FindRoute must select one of the the Y buses on the right side.
This only works if the interconnection are reserved for routing. In your scencario the PXI-2569 SPSTs in the middle must be reserved for routing.

The time for finding the route then rises to about 1.5s to 2s due to the additional route reservations.

You should see that long Route Finding for loopbacks as well.

Best Regards
Matthias

[PePf_TS]

Hi Chad,

thanks for your quick help and deep view into Switch Executive.

We are the OEM using the Pickering Switch Matrixes, controlled by the Switch Executive. In fact, we have a more complex setup of 12 Switch Matrix units. And we have the requirement from our customer to setup ~400 connections within 4 seconds. That means to find a route and to set it up within 10 milliseconds. Within a simple setup this is nearly possible. But in a complex system, the NI Switch Executive is too slow to find a route. In our setup it takes up to 44 seconds to find a route. Just too far away from the customer’s requirement.

But in my opinion there is one really bad behaviour of the Switch Executive consuming seconds of a routing calculation. Our configuration looks like:

To configure that in Switch Executive you need to hardwire the pins comF1, xE1 and chD1 together. This is ok. But now, to setup a routings it is required to set all the hardwired pins (comF1, xE1 and chD1) to “Reserved for Routing”. So they are hardwired and there it is not necessary to set the “Reserved for Routing” three times.

Why is that a problem? In our configuration we have 8 matrix units connected over the Y channels (600 x channels over 4 Y channels). At the first 4 units we have 4 units connected to the X channels in parallel to get  300 X channels over 8 Y channels. That results in 908 hardwire and 1248 “Reserved for Routing” settings within one configuration. And this is the reason for the huge time consuming routing calculation of the Switch Executive. If the “Reserved for Routing” can be set to one hardwired connection, instead to the three pins of that hardwired connection, this will reduce 600 “Reserved for Routings” settings. That means a routing calculation time of around 8 seconds, as of 44.

Well, all that does not help us. The Switch Executive is written in LabView. And this is not really performing. A quick C based tool can drive it. So we changed to that.

But thanks for your help.

Best regards,

Peter

 

[JasonWhite]

Peter,

I agree with your assessment that taking seconds to find a route in this particular configuration is not good behavior.  I thought that I would pass along a few pieces of info for you, though, just to correct any misunderstandings.

First, the core of Switch Executive is not written in LabVIEW but rather in optimized C++ code.  (I should note that I'm not trying to imply that LabVIEW is slow...a speedy LabVIEW application could also be written to handle this routing if done properly...but we used C++ in our case due to some other design requirements and our ability to optimize some of the memory management)  We do leverage LabVIEW for some of the configuration in MAX but it is purely for GUI.

Second, for your particular application, the best option is likely going to be either a) using a preconfigured route or b) manually creating a route string at run time.  Both of these options may not meet your end goal of trying to be more dynamic at runtime but it is the best that we can offer at this time given your hardware/software constraints.  The reason that the routing is taking so long is that once Switch Executive finds a route through your system, it looks at the number of relays that it is going to have to traverse and tries to find a more optimized path that would use fewer relay resources.  Due to the number of channels in your system and the way that they are configured, this search ends up being exhaustive and that is what is taking so incredibly long.  Switch Executive does have optimizations to greatly improve that speed if the topology were implemented by exposing multiple devices with hardwires.  As we look to the future, we could consider adding some additional options on the routing algorithm so that users could give some input into their routing.  For example, we could give users the option to optimize for minimal resource usage, optimize for fastest routing (ie. take first route), or supply a channel count that represents the shortest path so that Switch Executive will know when to stop searching since it has found the shortest route.

Thanks for all of your detailed feedback.  

Jason 

[PePf_TS]

Jason,

thanks for your reply and the good information.

First, good to hear that the Switch Executive Core is not based on LabView (sorry, was my fault). But I am still a bit confused why a simple routing calculation on a 75x4 (only one matrix, no isolation relais) matrix  takes minimum 10ms on a fast Core I7 workstation. This should be possible in less than 30us. Any idea?

For the second part, ok, I agree. Our application is a challange for every routing calculation program (but challanges improves the world, isn't it?). Please think about it if there is a possibility to improve the Switch Executive in such a way, to combine the hardwired and "Reserved for Routings" channels to one channel. This should help a lot in such complex configurations.

Peter

[Now_With_Underscores]

Hi Peter,

This is because the IVI standard doesn't actually know the topology of the switch when finding a route.  Thus, each time we hook R2 to R7, we can't zoom out and see that R2 is a row and R7 is another row that connects via any column and we simply need to connect any two relays on the same column.  Instead, we must look at what IVI does give us: R2 can connect to C1, can C1 connect to R7?  The answer is yes, so after we look to see that, then we look and see if there's a better connection.  so we see that all the columns that aren't in use will work.  This doesn't take a lot of time on one switch, but when we start expanding the number of lookups we must perform becomes quite long.  The reason we do this is so that we can find the best route (usually indicated by the least number of relays) to preserve the integrity of the signal.  

Adding an option to use the first available route would drastically increase the search time, but might not find the best route.  I'm recommend creating a product suggestion if this is a feature you'd like to see (http://digital.ni.com/applications/psc.nsf/default?OpenForm&temp1=&node=)