09-19-2008 04:46 AM - edited 09-19-2008 04:47 AM
09-19-2008 12:22 PM
Ryan
Thanks for drudging through those text files!
You should also know that the problem is intermittent, possibly explaining why it worked for so long and only failing after putting in a new PC.
Day before yesterdat we restarted the system cold and as of yesterday afternoon all sites were communicating fine with only 12 or so errors and no errors on the RS485 connected PLC. That RS485 channel was our main clue, since it has never had any errors with 10s of millions of transactions before this latest problem. I also have a gut feeling that newer motherboards without onboard UARTS and UART clocking harware for serial ports are a little (a lot?) more unstable with RS232 comms. We never had to restart Lookout this much with the older Intel hardware so that's where the gut feeling comes in, so YMMV.
Thanks again
Roger
09-24-2008 11:06 AM
Polling stopped at 3am this morning. Owner could not get lookout to start polling again even after severeal attempts to restart system.
I noticed that even on a good day, it takes Lookout 3 minutes to start polling the first site after a restart. I haven't seenh that behavior before.
Today I will wipe the HDD, reinstall XP and Lookout. Wish me luck....
09-25-2008 04:50 AM
Roger,
I sent you a fix I made on this problem. It is not an official release, but maybe you are interested in trying it first. We will do more tests before publishing it on web.
09-25-2008 10:05 AM
Ryan
Thanks! I will give it a try as soon as the new industrial PC arrives (no more internet/gaming PCs arrrrggghhh) which will be on Monday at the latest.
When I arrived at the site yesterday, Lookout had finally started polling correctly but only after several hours of being "stuck" on the first site. I instructed the owner on how to close the other processes until the first was done and open the other processes one by one as a last resort self help routine.
We also did the following changes to the system:
Removed the link between C5 and C4 where Lookout was providing C4 with target data for C4s operations from C5. We installed a high reliability peer to peer wireless network to handle this. That keeps Lookout from writing to C4 as C5 is being read and will cut down on traffic.
I also removed the sensor scaling screens that had 16 or so bi-directional flaoting point transactions. This had been recommended by IRTC at Cal Poly but isn't really needed ususally since if scaling needs to be modified it is best to do that with a laptop at the site. Scaling parameters are done in SCADA Pack logic.
In the end we cut the I/O from 410 to 356.
Haven't had a report back yet.
Thanks again for the new cbx!
Roger Foote
Foote Control Systems
10-03-2008 12:27 PM
Well, just composed a page long post that timed out and got deleted when I hit submit. When will I ever learn to copy my work to the paste buffer before hitting submit?
Long story short, the new cbx was installed Tuesday afternoon on the ag system and a small simple muni system and comms have been solid since.
The ag system is on a new industrial PC with PCI serial ports and the muni system is on a new intel mini tower system with a USB>serial converter.
The ag system uses R Domers sequencer idea which now is working beautifully, the muni uses simple PollRate of 20 seconds. Both showing 60,000 or so transactions with 3 or 4 errors which is totally acceptable.
By Monday we will feel secure that the problem we had is in the past.
Thanks to all that worked on the new modbus extension from us and our customers.
Roger
10-06-2008 06:36 PM
Well, we still have close to 100% good comms since the modbus patch.
We still see a situation occur where Lookout will repeatedly stream to a site for a long time, but at least now, it stops after a while and normal polling can continue.
It would be really good to find the bug that is causing this since when it happens it does really slow the throughput down. It happens several times per 24 hours.
Is there a chance you can work on this?
Thanks in advance
Roger Foote
10-06-2008 10:28 PM
Roger,
How long? 1 minute or 5 minutes?
I don't think this is a bug. As I said, the modbus driver update all the outputs every 100 polls by default. If one poll cycle takes 30 seconds, there will be about 30 times to update all outputs every day. I checked the diagnostic file you sent me. Each site needs about 30 seconds to update outputs. During the updating, if it meets communication problem, it will retry. By default, it will retry 4 times. So, it's possible that the updating outputs on one site takes more than 30 seconds, even more than 1 minute.
Could you make a diagnostic file again for me to check it out? Please also take down the approximate time when it repeatedly polls a site and how long before it stops, so that I can easily locate it in the file.
By the way, you can set how often to update the outputs. You can change 100 to 200 or more. I guess that if you set it to 50, you will see it more often work on one site for long time.
10-06-2008 10:34 PM
10-07-2008 09:10 AM
Hi Ryan
I will check to see if this is happening every 100 Polls, but one thing I can tell you for sure is that it happens every time there is a break in communications.
I have to break comms manually since radio comms are so good at this site. But every time comms are interrupred, it polls the SCADA Packs 70 or so frames before settling back to the normal 12 frame polling cycle.
But, we actually have every object in the ag system I sent you set up as bi-directional connections. I think I mentioned that we have OITs at every site.
Changes in Lookout get updated on the OITs and changes in the OITs get updated in Lookout. It would be good to turn off the "Update outputs every XX polls" feature for these systems where Lookout always knows what the PLC state is.
Your help is very much appreciated!
Roger