Periodically losing serial communications - LabVIEW 8.5/DSC module

[MReveley]

I am running a chemical process using a LabVIEW 8.5 interface in Windows XP. All continuous-range controllers are sent setpoints via serial communications set-up through a field-mounted ENET 485 unit. I set up a separate modbus I/O server for each of the two COM ports connected to Modbus-compatible controllers, and have successfully initiated control with all the units. However, every once in a while (around once a day, with no predictable timing), I lose serial communications. My ENET 485 shows active connections on the ports, and the variable manager shows activity in all but one of the shared variables (it varies which one crashes), however my main VI drops all values from the shared variable engines and reads zeroes. I have my program set to undeploy the variables programmatically at program stop, and this is when I get an error message:

"Error 1967353906 occurred at shared variable write in TVCControl.vi

LabVIEW DSC: Hex 0x8ABC8FCE

The response from the modbus slave device is not readable. Check the communication settings between the modbus master and slave devices."

Typically undeploying the variables (without errors) turns off the communications lights at the ENET unit, but when I get this error, these indicators stay lit after the program has stopped. I have to manually undeploy all the libraries before I can close communications and restart the program. This is a serious problem, as this interface will be controlling a chemical process that needs to be stable for weeks at a time. I have been in touch with tech support for the past few weeks, but have not as of yet discovered the problem. Does anyone have any ideas?

Matt

[chris_amadasun]

Lets try and determine the cause of this issue.
Can you take some NI SPY logs.
Navigate to Start >> Program Files >> National Instruments >> NI SPY
Under Tools click on Options and set the log length to around 100 lines, once you get the error, please save the log file,
post the log file here.

[MReveley]

I will attempt to do this; unfortunately, the error only happens once a day or so, often when we leave the program running overnight. Is there any way to access a larger backlog of entries in NI Spy if we miss the error by several hours? (right now it is running about 9000 lines/min or ~500k lines/hr) If not, we just have to wait until either the communications fail when the process is running or if the error happens by chance during the day while we are monitoring it. I will keep you posted.

In addition to the failure of the Modbus engine, I also periodically lose communication with several mass-flow controllers I am running through a program called SmartDDE, communicating with LabVIEW via DDE. Unfortunately, I can't tell whether the failure is happening in LabVIEW, in the DDE conversation link, in SmartDDE, or in the hardware interface. Any thoughts?

Both of these errors occur rarely, but are extremely problematic, as this process needs eventually to be able to run steadily for 4-5 days at a time.

Matt

[chris_amadasun]

Hello,
We should probably check the NI Spy logs.
If you open NI Spy and navigate to Spy>>Options
In the General tab you can change the number of lines in a single log.
In the Logging tab you can specify that NI Spy log everything and spill onto multiple files.
In this case the files of interest would be the last ones, since these happened before the error.
(Please see attached image)

[MReveley]

Christian,

I was able to catch the errors in NI Spy. I am posting excerpts from the files now. Loss of modbus serial comms happened at 11:15:29 PM in the file "capture.000219". There is a line in NI Spy that is colored red (line 30375), I imagine this is the error that occurred. Loss of DDE communications happened at 6:51:39 PM in the file "capture.000151". There was no particular line in this file that stood out to me. Please let me know what you think. It could have happened a second or two before, but not much more than that.

Thanks!

Matt

[chris_amadasun]

Hello Matt,

I have checked the file "capture.000219.txt", I was not able to open it in NI Spy but I did dig around in wordpad.
I could not see any errors.
This may be an inconvenience but I do need spy log capture. This time when you take the spy log could you click on the save icon instead of copying and pasting to a txt file?

What I can deduce from the file is that you are communicating through two ports on your ENET 485, I want to know in your LabVIEW application if you are using shared variable to
for the VISA resources?

Thank you.

[MReveley]

Christian,

What did you think of line 30375 in the text file as I pointed out before? It reads:

> 30375.  viRead (ASRL5::INSTR (0x01725028), "...", 3, NULL)
> Process ID: 0x00000160         Thread ID: 0x000001A4
> Start Time: 23:15:29.281       Call Duration 00:00:00.000
> Status: 0xBFFF006B (VI_ERROR_ASRL_FRAMING)

This line was also colored red in NI Spy, and was the only line out of 30,000 in that file that was red and read "error" somewhere in it, and happened at the exact second we lost serial communications. I can't post the NI Spy file on the forum because I think it is too big (~10mb). If you give me your email address, I can email it to you.

Matt

[chris_amadasun]

Hello Matt,
That error is a frame error, and is usually caused when the instrument and device have mis-matched baud rates.
The following KB discusses that error code: Frame Errors

NI Spy has an option to save a certain number of lines for viewing, or you could split the logging across multiple files.
For now you could upload your spy log here. Please let me know when you have uploaded it to that directory.

Thank you.

[MReveley]

Posted "capture.000219". Let me know what you think. What would trigger that frame error after the process had been running steadily and communicating well for 2 days? Could I be trying to send and receive with too short a delay?

Matt

[chris_amadasun]

Hello Matt,

Thank you for the spy log. Whenever you perform a read there is a warning 0x3FFF0006.
It might be that you are not reading all the data in the buffer. In your application how much data do you read in during each VISA Read?
You can also using the Bytes at Port property in LabVIEW and wire that to the VISA Read.

In the spy log I did not come across any red lines, this time around there were probably no Frame Errors.