Lookout

cancel
Showing results for 
Search instead for 
Did you mean: 

Lookout 6.0.2 modbus rtu problem?

Solved!
Go to solution
Roger,

Looking into the diagnostic file and the source code, I have found the problem. Yes, it is a bug in Lookout modbus driver. Actually, the only polling object doesn't poll for multiple times, but just for once. Other modbus objects don't stop, but just wait for the polling object.

Modbus driver has a feature that periodically refresh all the outputs in order to keep consistent between lookout and hardware. The problem happens during the refresh. By default, the refresh is executed every 100 polls. It writes all outputs one by one, if one write command fails, it will restart the refresh from the very beginning. This is where the problem happened on your site. The communication seems not good, so the refresh always got wrong response, and restart the refresh from the beginning, then failed again, redo.... Only when all the outputs are refreshed without problem, it is done. So, this object takes a long time to finish this poll when it always gets wrong response. The logic in the source code I think has something wrong.

I will try to fix it as soon as possible.

Btw, take a look at this KB. http://digital.ni.com/public.nsf/allkb/2E64D5CF87CA6A1086256BB30070DC1A?OpenDocument
If you increase the RefreshOutputs to 200 or higher, the probability of the problem will be decreased. I'm thinking to disable the refresh can be a workaround, but this may cause some other problems. If you don't refresh the outputs periodically, the value on hardware may not be the same as the one in Lookout, so it's not good to disable it.
Message Edited by Ryan.S on 09-19-2008 04:47 AM
Ryan Shi
National Instruments
0 Kudos
Message 61 of 89
(2,834 Views)

Ryan

Thanks for drudging through those text files!

 

You should also know that the problem is intermittent, possibly explaining why it worked for so long and only failing after putting in a new PC.

Day before yesterdat we restarted the system cold and as of yesterday afternoon all sites were communicating fine with only 12 or so errors and no errors on the RS485 connected PLC. That RS485 channel was our main clue, since it has never had any errors with 10s of millions of transactions before this latest problem. I also have a gut feeling that newer motherboards without onboard UARTS and UART clocking harware for serial ports are a little (a lot?) more unstable with RS232 comms.  We never had to restart Lookout this much with the older Intel hardware so that's where the gut feeling comes in, so YMMV.

 

Thanks again

 

Roger

 

0 Kudos
Message 62 of 89
(2,822 Views)

Polling stopped at 3am this morning. Owner could not get lookout to start polling again even after severeal attempts to restart system.

 

I noticed that even on a good day, it takes Lookout 3 minutes to start polling the first site after a restart. I haven't seenh that behavior before.

 

Today I will wipe the HDD, reinstall XP and Lookout. Wish me luck....

0 Kudos
Message 63 of 89
(2,793 Views)

Roger,

 

I sent you a fix I made on this problem. It is not an official release, but maybe you are interested in trying it first. We will do more tests before publishing it on web.

Ryan Shi
National Instruments
0 Kudos
Message 64 of 89
(2,783 Views)

Ryan

 

Thanks! I will give it a try as soon as the new industrial PC arrives (no more internet/gaming PCs arrrrggghhh) which will be on Monday at the latest.

 

When I arrived at the site yesterday, Lookout had finally started polling correctly but only after several hours of being "stuck" on the first site. I instructed the owner on how to close the other processes until the first was done and open the other processes one by one as a last resort self help routine.

 

We also did the following changes to the system:

 

Removed the link between C5 and C4 where Lookout was providing C4 with target data for C4s operations from C5. We installed a high reliability peer to peer wireless network to handle this. That keeps Lookout from writing to C4 as C5 is being read and will cut down on traffic.

 

I also removed the sensor scaling screens that had 16 or so bi-directional flaoting point transactions. This had been recommended by IRTC at Cal Poly but isn't really needed ususally since if scaling needs to be modified it is best to do that with a laptop at the site. Scaling parameters are done in SCADA Pack logic.

 

In the end we cut the I/O from 410 to 356. 

 

Haven't had a report back yet.

 

Thanks again for the new cbx!

 

Roger Foote

Foote Control Systems

0 Kudos
Message 65 of 89
(2,779 Views)
Solution
Accepted by topic author rfoote

Well, just composed a page long post that timed out and got deleted when I hit submit. When will I ever learn to copy my work to the paste buffer before hitting submit?Smiley Tongue

 

Long story short, the new cbx was installed Tuesday afternoon on the ag system and a small simple muni system and comms have been solid since.

The ag system is on a new industrial PC with PCI serial ports and the muni system is on a new intel mini tower system with a USB>serial converter.

 

The ag system uses R Domers sequencer idea which now is working beautifully, the muni uses simple PollRate of 20 seconds. Both showing 60,000 or so transactions with 3 or 4 errors which is totally acceptable.

 

By Monday we will feel secure that the problem we had is in the past.

 

Thanks to all that worked on the new modbus extension from us and our customers.

Roger  

0 Kudos
Message 66 of 89
(2,719 Views)

Well, we still have close to 100% good comms since the modbus patch.

 We still see a situation occur where Lookout will repeatedly stream to a site for a long time, but at least now, it stops after a while and normal polling can continue.

 It would be really good to find the bug that is causing this since when it happens it does really slow the throughput down. It happens several times per 24 hours.

 

Is there a chance you can work on this?

 

Thanks in advance

 

Roger Foote

0 Kudos
Message 67 of 89
(2,684 Views)

Roger,

 

How long? 1 minute or 5 minutes?

 

I don't think this is a bug. As I said, the modbus driver update all the outputs every 100 polls by default. If one poll cycle takes 30 seconds, there will be about 30 times to update all outputs every day. I checked the diagnostic file you sent me. Each site needs about 30 seconds to update outputs. During the updating, if it meets communication problem, it will retry. By default, it will retry 4 times. So, it's possible that the updating outputs on one site takes more than 30 seconds, even more than 1 minute.  

 

Could you make a diagnostic file again for me to check it out? Please also take down the approximate time when it repeatedly polls a site and how long before it stops, so that I can easily locate it in the file.

 

By the way, you can set how often to update the outputs. You can change 100 to 200 or more. I guess that if you set it to 50, you will see it more often work on one site for long time.

Ryan Shi
National Instruments
0 Kudos
Message 68 of 89
(2,676 Views)
By the way again, it needs to update the outputs because you make the one-way connection. In Lookout, for example you set modbus1.AAA = BBB. When BBB changes, its value will be sent to modbus. But it is one-way connection. If AAA in PLC changes by itself(maybe it will not happen), BBB in Lookout doesn't know. So, we need to periodically write BBB's value to modbus. By default, every 100 polls the modbus writes all this kind of outputs to modbus slave one by one, in order to keep consistent between the control in Lookout and the value in PLC.
Ryan Shi
National Instruments
0 Kudos
Message 69 of 89
(2,674 Views)

Hi Ryan

 

I will check to see if this is happening every 100 Polls, but one thing I can tell you for sure is that it happens every time there is a break in communications.

I have to break comms manually since radio comms are so good at this site. But every time comms are interrupred, it polls the SCADA Packs 70 or so frames before settling back to the normal 12 frame polling cycle.

 

But, we actually have every object in the ag system I sent you set up as bi-directional connections. I think I mentioned that we have OITs at every site.

 

Changes in Lookout get updated on the OITs and changes in the OITs get updated in Lookout. It would be good to turn off the "Update outputs every XX polls" feature for these systems where Lookout always knows what the PLC state is. 

 

Your help is very much appreciated!

 

Roger

0 Kudos
Message 70 of 89
(2,667 Views)