Visa timeouts w/ multiple serial devices

[danielbixby]

Hello,

I am using NI-Visa 5.0 to communicate with multiple power supplies via Enet-to-RS232 gateways. There are multiple serial ports per gateway, and each power supply is connected to one of the ports. Each equipment model process within our software connects to a single port and periodically queries status for the power supply on that port (about every 20 seconds).

When testing our software's response to a disconnect, we encountered a strange phenomenon. Here's an example:

Gateway #1: Ports 1, 2 connected to power supply A, B

Gateway #2: Port 1 connected to power supply C

The Ethernet plug was pulled from Gateway #1. As expected, we soon received timeouts for the processes connected to Gateway #1 and monitoring supplies A and B. What is strange is that, soon after, we also received timeouts for the process connected to Gateway #2. This doesn't make physical sense, as the processes are talking to completely separate devices.

We've tried increasing the Visa timeout values for these sessions, and looked into a few other Visa attributes, with no success. Only if we monitor Gateway #2 from a separate host is the process unaffected by the timeout/reconnect activity related to Gateway #1. 

It seems as though the initial timeouts, and subsequent reconnect attempts, through Gateway #1 are interfering with the software's ability to read data from the device off of Gateway #2. Is it something that is shared at the Visa level? Has anyone encountered anything similar to this? 

Thanks in advance

- Dan

[Tim_O1]

Hi Dan,

That is odd that the two gateways are interfering with each other. Can you post the code that you are using to run these two gateways? What is the model number for the gateways and any other hardware you are using? Do you get any error codes?

Tim O

[danielbixby]

Hi Tim O,

Thanks for the response. The gateways are National Instruments ENET-232/4 Serial Device Servers. The power supplies are a variety of Sorensen/Ametek devices (e.g. HPD 30-10, XT 30-2, XT 20-3).

A few details on the code. It runs on Redhat 5.5 (g++). Each process uses the standard Visa functions to communicate with the devices. So you have the usual viOpen, viClose, viSetAttribute, etc. Every 20-30 seconds the process polls its corresponding devices by using viWrite to send the query (e.g. "VOLTAGE?") and then viRead to get the response string. The error code received is VI_ERROR_TMO, both for the gateway whose plug was pulled and then for the other gateway(s) that are still connected.

Here are the Visa attributes that we currently have set:

VI_ATTR_ASRL_BAUD 9600

VI_ATTR_ASRL_DATA_BITS 8

VI_ATTR_ASRL_PARITY 0

VI_ATTR_ASRL_STOP_BITS 10

VI_ATTR_ASRL_FLOW_CNTRL 0

VI_ATTR_ASRL_END_IN 2

VI_ATTR_TERMCHAR 0xD

// VI_ATTR_TERMCHAR_EN 1 << same results whether this is set or not

The Visa timeout attribute is allowed to default to 2000 (msec). As mentioned before, we tried increasing that value with no success.

Hope this helps for starters.

Thanks

Dan

[Tim_O1]

Hi Dan,

It seems like there could be a few different things going on here. Could you post a screen shot of your LabVIEW code?  I think one cause to your issue could be something like error handling where as soon as one gateway goes down it sends that error though the other gateway in LabVIEW which causes that one to go down. Below are a couple links that may help you.

ENET 232 Manual (page D-1)

http://www.ni.com/pdf/manuals/370323b.pdf

Serial Port Communication

http://zone.ni.com/reference/en-XX/help/371361H-01/lvmeasconcepts/serial_port_communication/

Tim O

[danielbixby]

Hi Tim O,

I'm not highly familiar with LabVIEW - mainly because we're not using it. Rather, we're simply running our own software, where the processes utilize the Visa 5.0 libraries and have a 1-to-1 mapping with the devices on the other side of the Enet-to-232 gateways.

Thanks for posting the documentation links, I printed them out and will have a look. In the meantime, I thought I would further simplify the original scenario based on what I saw in testing.

Gateway #1: Port 1 connected to power supply A. Process #1 on the Linux host connects to this port.

Gateway #2: Port 1 connected to power supply B. Process #2 on the Linux host connects to this port.

1. Gateway #1 is disconnected from the network by pulling the Ethernet plug.

2. Process #1 attempts to query Pwr Supply A's status - using viWrite() and viRead() through Gateway #1. Visa returns the VI_ERROR_TMO code, which is expected.

3. A few seconds later Process #2 attempts to query Pwr Supply B's status through Gateway #2. Visa returns the VI_ERROR_TMO code even though that gateway is still on the network.

It's almost like the 2 processes are sharing some resource that gets affected by the initial disconnect, then somehow propagates his error to the 2nd process. Perhaps there is some setting I need to use when opening the resource manager? It's getting low-level I know. Compounding the strangeness is that we've had a similar setup with TCP/IP devices; however their disconnects and reconnects never appeared to interfere with each other.

Thanks again

Dan

[Tim_O1]

Hi Dan,

What language is this written in? Can you go though the code and debug it to see which call exactly causing the timeout? Try to see if the code is hanging on Gateway #1 when its unplugged and see if any calls are being made to Gateway #2 then. I know you said they are on two processes but step though the code and make sure that is true and find what function is causing the timeout.

Tim O

[danielbixby]

Hi Tim O,

As mentioned, our code is written in C++ and runs on Redhat 5.5. I studied the log files some more and have a few more details.

After Gateway #1 was unplugged, the next call by the corresponding process to viWrite() resulted in an error (believed to be VI_ERROR_TMO) that caused the process to close the session. The process continued trying to send queries every 20-30 seconds. For each query, it first detected that the session was closed and thus attempted to re-open it with viOpen() prior to sending the command. The re-open would obviously and perpetually fail since the gateway was disconnected (VI_ERROR_RSRC_NFOUND), and thus not bother sending the command.

For Gateway #2, I zeroed in on the timestamps of the errors that occurred after Gateway #1 was disconnected. Interestingly, each error appeared to occur at the same time that our Gateway #1 process was trying to auto-open its session. These errors always came back from the call to viRead() - sent after a successful viWrite() - and with a code of VI_ERROR_TMO.

After a few minutes, the Gateway #1 process was shut down and thus ceased its reconnect attempts. The Gateway #2 process then continued without any more errors.

I hope these details help. The equipment was recently taken away for testing, which is why I've had to rely on the log files from the last test run. It should however be back in our lab in a couple of weeks, at which time we can run more detailed tests.

Thanks again,

Dan

[Tim_O1]

Hi Dan,

Thanks for the details. Since Gateway#2 works fine after Gateway#1 shuts down, it looks like the errors from viOpen() on Gateway#1 are causing the problems with Gateway#2. Can set your code to handled the VI_ERROR_RSRC_NFOUND error? You could stop Gateway#1 from auto-opening its session or try to write an exception for the error. This will allow the viWrite() call on Gateway#2 to work without any errors I believe.

Once you get your hardware back, play with how viOpen() is called and handled on Gateway#1 and you should be able to get everything working hopefully.

Tim O

[danielbixby]

Tim O,

Unfortunately, we have a requirement for our software to periodically attempt reconnection once it loses a bus session.  The phenomenon of viOpen() calls to one device interfering with communication to a separate device is of course a new one for us, despite a history of controlling multiple devices via corresponding processes on a single host. I had hoped that it was something simple like a Visa attribute we were or weren't setting, or even an existing fix in a later version of Visa. But it sounds like this is a new issue on your end as well?

Thanks again for your responses. Please let me know of any additional questions/suggestions.

Dan

[Tim_O1]

Hi Dan,

Unfortunately I still can not find any Visa attribute that you are missing that would simply resolve this issue. It does look there may be an issue with viOpen causing interference when another device goes down. The only other thing I could suggest would be take a look that the VISA manual(link below) and look at the Multiple Interface Support Issues it may have something that you missed.

http://www.ni.com/pdf/manuals/370423a.pdf

Good luck to you and I hope you can get it working soon,

Tim O