I'm setting up a system to test the modbus. Could you give me the customer's process file? If possible, include the lookout.ini file. I may not use the wireless modem, but I hope to run the same process. You can upload it to ftp://ftp.ni.com/incoming. Also tell me an Email address to access you. I can give you an expiring development license for your local install.
We will try our best to help you and to improve our software. If we reproduce the problem here, even the bad communication after 24 hours, we can look into it and fix it. But just to think and look at the source code, it's hard to figure it out.
Btw, the files on your system are correct. They are officially released 6.0.2.
Thank you for your advice. It's a good way to notify the operator when the communication fails. Although the Roger's application displays alarms, it's not acceptable to have the communication stop. The modbus objects seem to go into a bad status, no polling, and no retry. We need to find the root cause and fix it.
I am still pieceing together the exact details of what alatms caqmethrough but it looks like the ag system got none. Looks like the municipal system got one com alarm and one pump alarm through a Verbatim dialer. The operator ignored the one pump alarm because the system has 3 wells and the others should have taken over. They did not and did not issue any dial outs. The ag system uses pager objects, and they did not work either.
The ag system is 15 or 16 processes.
We are going to be on the road for a week or so deploying yet another Lookout license and I won't be able to do anything until the new system is on line.
Sorry for the confusion and all. We just got released from the evacuation threat of the wildfires that were within 1 mile from our house. We were supposed to start the job we are on around 7/1 but the USFS had other plans. Btw, 1/3 of Trinity County has been burned in fuel load burnouts since 1999. We now need to get compensated for the $30,000 in materials that were paid for on 7/11.
Lots of the info in my thread were collected late at night between fire threats. All of our records are still in boxes ready to go.
Again, after phone conversations yesterday the alarm situation is that the municipal got 1 com alarm and 1 pump alarm before the modbus object stopped polling the Verbatim dialer.
The ag customer got nothing (no dialer there just pager objects, 24 of them IIRC).
The municipal was the first failure and I honestly thought that the computer had crashed, so I had them reboot win XP. I pretty much gave it no other thought after that until the ag system did the same thing and even then I didn't put it together until I was posting here. As I said this all was happening in the middle of the fire incident, sorry for the confused irritated demeanor.
This has been a tough 2 months.
I will ask the ag system manager if he will release the files.
Hope it is clear now.
Yes, it's much clearer now. FIrst, the RKdomer's idea is good. At least when the Modbus stops polling with no reason, the operator can get an alarm, then as you said, to restart Lookout. This is a very good defence, and a workaround for now. I think it seems like the Modbus object is idle, all the addresses keep the last value. The Modbus doesn't send out command, that's why there is no communication alarm. This idle status is what we need to figure out.
You mentioned the crash, I want to confirm with you that it was not a crash or hang. If Lookout really crashed, it's another issue.
Btw, if the customer will release the files, I will keep it confidential.
Ryan On that muni system I assumed it was a crash, either windows or Lookout and just had them reboot the PC. Then the same type of failure happened on the ag system. I posted and that is when I figured out that it was the same failure, the one where the polling stuck on one site. It helped that the manager of the ag system is familiar with Lookout having taken the classes on Lookout offered by Cal Poly, Chico through our freinds at the Irrigation Technology Resource Center. After talking to him on the phone and having him actually watch the modbus statistics screen showing the polling stay on only one site, I knew what had happened.
I will try to upload the municipal file, since I have been delayed on my road trip until early next week. That process used Koyo DL05s and would be somewhat reproducible.
The ag system is extremely complex with many subnetworks after the remotes being polled by SCADA Packs. It is also complicated by the fact that there is dual HMI at all the main sites. Lookout and SCADA Pack Vision 50 OI screens that both update when changes are applied to either HMI. Lots of bi-directional traffic being a "distributed control" system, which also menas in this case that ALL logic is performed in the SCADA Packs not in HMI, per Cal Poly's reccomendations.
The muni system is "centralized control" where all logic is performed in Lookout and the RTUs (DL05s) are "dumb", only reporting 1 or 2 analogs and acting as DO to start pumps.
You may need to change the data members if you use other PLCs or modbus slaves to simulate the remote sites in the system system.
I have set up and run your process.
Btw, when the problem occured, did you or the operator notice if totol errors was increasing?
If the total errors was increasing, it means the modbus was still active, but no response. But if not, the modbus objects were not working.
The error count did not accumulate on the sites that stopped polling. Modbus was not idle, just hammering away at one site in a loop, never returning to the other modbus objects.
The extra alarm layer that R Domer suggested is excellent and I will try to implement it in my systems as I go, possibly I can make the rounds to my clients this winter for a "maintainance upgrade" to their systems.
Question: I realize that 1 a second interval on the polling keeps things a little busy but on the ag system it takes hundreds of transactions to complete a polling cycle. That being said, do I really need to increase polling time? 2 seconds enough? Or does Lookout need more time than that?
Lookout doesn't need much time to execute the codes, maybe milliseconds is enough. But the response from remote takes time, especially when there is a large number of points to poll. For example, there are several devices, Lookout needs to poll each one for tens or hundreds of points, this whole cycle may take more than 1 second. Lookout needs to send command for reading one or a batch of points, and wait for reponse, then send command for the next several points. So, it's possible that the system you said actually takes more than 1 second for each polling cycle, then you will see that Lookout is busy since it is always polling and no much space to do other things.
The process you uploaded is simple and doesn't read a lot of points. From the diagnostic file, I see that each polling cycle takes about half second. So if there are hundreds of transactions, it will probably take more than 1 second to complete one cycle. Then, even if you set it as 1 second, the actual rate might be different.
It's hard to say if 2 seconds is enough, it depends. For example, if it actually takes 2.3 seconds to complete one cycle, set it as 3 or more seconds may be proper.