Lookout losing data and sending alarms when no alarm condition exists

[rfoote]

We have a PLC controlling 8 pumps to keep a tank full. The PLC, a SCADA Pack, does all logic.

We read this tank level back in Lookout.

Occasionally, the trend for this tank level drops to zero for 4 to 5 minutes causing an alarm to be issued and sending a page to the operator.

This dropout is long enough to start all 8 pumps as if the tank was empty, but no pumps are started and when the data in Lookout is restored the trend line returns to the previous value.

The other behavior we have noticed is that when Lookout is shut down and re-launched, a low tank alarm is logged for this same tank before Lookout even starts polling any sites. It takes 5 or 6 minutes for the polling to get to that site btw. This alarm happens immediately every time Lookout is re-launched.

Could this be a corrupt database?

We are on Lookout 6.0.2 running under Windows XP Pro on an Intel motherboard with a single processor.

Any insights welcome!

Thanks

Roger

Foote Control Systems

[Mike@DTSI]

The level going to 0 could be caused by a bad transducer, analog input or possibly even communications.

Lookout saves the last state of the level in a state file.  If the level has a math conversion done in lookout, sometimes lookout sees a 0 when it first starts, causing this same problem.

We add a startup delay on timer to prevent level conversion when lookout first starts until the sites are polled no operations take place.

Mike

[rfoote]

Thanks for the suggestion, Mike.

But, if you re-read the original post we have eliminated that possibility. Pumps have a 10-40 second start delay in the PLC, the event lasted 5 or 6 minutes and no pumps started.

All 8 pumps will start if the transducer, a Foxboro IGP10, failed for more than the pump start delay times.

It turns out that it was a database problem. In the original post I mentioned the alarms showing up as soon as Lookout was launched... That was a corrupted database and Lookout was using that zero value alternately from what I can gather.

The database was so corrupted that MAX could not delete or detach it. I had to stop all NI processes under Windows, manually delete the database folder, then let Lookout create a new database. Then using MAX I could delete / detach it if I wanted to.

Not sure what causes the database to become unusable to MAX but still function under Lookout, albeit in a error prone way... 

Roger

[Mike@DTSI]

Yea, citadel still needs some bulletproofing to compete.  The SQL Server useage helped a little but has has a long way to go.

Good to know it works now.

Mike

[rfoote]

Mike

True, more bulletproofing would be ideal.

What would be nice in the interim would be a health datamember for citadel to alarm the user when the db gets uhappy.

Still usintg that modbus fix that Ryan sent me btw, and have had far less problems with serial comms since then.

I really hate the idea of going to another HMI software, old dog / new tricks and all, so I am always happy to find ways of keeping Lookout running.

Best

Roger

[Ryan.S]

I'm not sure what's wrong in Citadel. If the first problem about the zero value on trend doesn't happen again, it's fine.

But if it still happens, I also think it is more like a communication problem.

Lookout gets the data from device, and logs to Citadel. The alarm is usually based on the realtime data from device. Even if you are not logging data, the alarm should also be generated when the trigger condition reached. So I guess, the data from device goes to zero, and lookout logs zero to database, that's why you see the zero on trend. Because of the low value of data, the alarm is generated. Maybe the remote transducer is good, but Lookout gets the wrong data or zero. The problem is mabye in Lookout driver object or the communication between driver and the remote device.

[rfoote]

Ryan

Please re-read my first post... The only entity to report the zero value was Lookout. The SCADA Pack would have started all 8 pumps if the tank would have been somehow evacuated. Those pumps did not start, and when the even was over with the tank level returned to it's original value. Exact value as a matter of fact since there was no demand at the time.

Comms were good through this period.

When Comms break down the trend does not go to zero, it just stops logging and introduces gaps in the trend line. It would be CHAOS if things reverted to zero values during breaks in comms.

This problem has been reported by other customers but we never found out what it was until this episode.

Also, why would I have alarms from Lookout at startup before any remotes have been communicated with, every time until I trashed the database?

Those alarms stopped happening with a new database.