okay, I will try to set up a system to do a stress test on the Modbus object.
One more question, have you seen this problem for multiple times recently? As it was running well for years, I'm curious about if anything changed before this happened, including the process, the parameters of object?
In the first case, the municipal water district, the problem started with their upgrade to 6.0.2 from 5.1 which had run without comm problems since 2003.
In the second case, the Ag customer, this was the longest period the system has gone without a reboot. That system had been in construction for a long time so it was getting restarted every couple of weeks, sometimes more often.
In any case I think Mike and RKDomer are right, sounds like a polling buffer issue since a restart fixes it every time.
I have lots of 6.0.2 licenses out there, but most are 2 or 3 sites with only a single packet (Analog level in 16 or 12 bit) being transacted. There is a LOT more activity on these 2 systems.
My customer with the ag installation has been perfotming some tests for me. Here are his findings:
> For the first 24 hours after a relaunch of Lookout the modbus stats window shows as low as zero up to 2 or 3 errors across the system.
> After 24 hours there are hundreds of errors and the errors increase after that.
It is interesting to note that even the remote PLC that is hardwired to the computer via RS485 on a different serial port starts showing errors after 24 hours.
This client had 6.0.2 for a long time without errors but lost a motherboard and we reinstalled Lookout on an identical PC. I used my more recent install disk when we couldn't find his old disk and called NI for activation... We are thinking this is when things started changing for them since as I said before we were working on this system every few weeks and restarting Lookout after adding new processes. The customer would have noticed the errors after 24 hours if they were happening before we reinstalled Lookout on the new PC (again identical to the old one)
This roughly coincides with the other municipality's upgrade from a 50 I/O RTO to 100 I/O FDS system. I used the disk that came from NI for that upgrade.
Now I am wondering if modbus.cbx was changed since the older insatll we originally did for the ag customer...
Is there any way to look inside the install tome to see what versions are in the install without actually installing it? I do that all the time with other software through special tome viewer apps but I can't find a way to do this with Lookout.
Sure hope that we can go back to using Lookout the way we did before, as in 5.1 or earlier 6.0.2. Customers are getting very nervous.
This basically has to be associated with the installs that were shipping just before 6.1?
Are you sure that the "old" Lookout install was 6.0.2, but not 6.0.1 or 6.0? If it was 6.0.2, I don't think the modbus.cbx can be different. Take a look at current installed modbus.cbx and lkworks.dll, tell me the version and modified date of the file.
Could you make a screenshot of the alarm, and make a diagnostic file for the serial port?
We have never used 6.0 or 6.01 since we were aware of the problems you had with those versions. We never use new Lookout versions until you get them working PROPERLY.
We don't just go around slamming these things in without some thought.
I will say again... The modbus object just stopped polling all but one site. It just polled one site for hours, never returning to the normal cycle until Lookout was relaunched. That means the problem IS Lookout.
I am not a beta tester. Seems odd that you won't get going on this without questioning me since as others have pointed out, the modbus problems have NEVER been resolved.
For me to go to the site and set a diagnostic file will be nearly a whole day including 4 1/2 hrs travel time. Geez, if serial comms or radios were a mystery to me I would not have the success rate that I do and that being since 1983 when I started working with and repairing radio based SCADA systems, which includes making systems work that major contracts left non functional. I wouldn't have chosen this occupation if I were that easily stumped.
Now if the integrators license would allow running for more than 180 MINUTES I could help you more, but that was NIs call and you have to live with it.
I would rather report back to my customers that you are not dragging your feet the way it seems....
Post the details of the first modbus.cbx that was part of the fix for 6.0.1 and the details of the cbx on the current install disk please. (Date of creation and size)
At least then we would know as opposed to guessing.
You are acting like this is some sort of hobby and we have all the time in the world to do your job. It is not a hobby and my customers expect a lot more from me than this and I deli\ver.
Time for you to do the same! Or is this the level of support you offer to those that don't buy your "Support" package?
Ryan,I think something Roger touched on and I will emphasize is that when the polling stops there is no notification from Lookout. No communication failure alarm, nothing to get anyone’s attention. In my experience with this problem, the diag file will only confirm that the polling suddenly stops - poll, response, poll, response, and then nothing. To prevent the lack of communication from being overlooked, I have added an alarm object for each modbus object which uses a delayon of the modbus update datamenber as its trigger. This way the operator will (hopefully) wake up and take action when the alarm goes off. It might sound like overkill, but there are other conditions in which communication can stop without an alarm besides the ‘it stopped polling’ scenario this thread is concerned with.
Looks like I misspoke on my first post... I thought they got comm alarm pages but looking at the codes for the pages, they were low reservoir level alarms from pumps not coming on since the data was NG coming from one of the sites that stopped polling. Actual data would have turned on pumps but did not since the actual data was unavailable to the system because the site providing that data was not polling and did not issue a com fail alarm from either my alarms or the system com alarm (modbus secondary alarm) neither of these alarms happened. If a municipal system has a failure that lets the system pressure drop below 20PSI, they have to report it since that condition could lead to backflow. Not good.
That is why I am against trying to reproduce it on a live system. We cannot let this happen again.
Btw, my install is 6.0.2 build 1 and the modbus cbx is 128kB created on 11/17/05 at 6:28PM. I have not been able to get the cbx info from either of the affected systems because of their remoteness. I have not been able to recreate this in 180 minutes of run time. Comms start bogging down after 24 hours but do not fail until some time later... Could be weeks or months before this lockup happens.
I am seeing visions of SCADA Packs doing all the polling and Lookout JUST doing HMI which defeats the purpose of Lookout for me because I liked the way modbus polling worked and the statistics screen. Well, when it worked.