What is causing Lookout to Lock-Up

[SCADADog]

I am running Lookout V6.6 on an HP desktop computer with Windows 7 64-bit OS.  Lookout is running in XP mode.  We are experiencing increasing frequency of the Lookout application locking-up.  Yes, not much of a SCADA system when this happens.  The OS System Manager shows Lookout "Not responding" and I have to reboot the computer before Lookout will operate again.

This is a simple SCADA system controlling two well sites with lead and lag pumps at both sites.  It seems that lockups may occur more frequently when two pumps are being turned off near simultaneously.  The computer uses a serial connection to a Zetron Controller talking to two Zetron RTU's in the field.

[Mike@Forshock]

Not running 6.6 here regularly so can only recommend on my own observations.

Symbolic links that user a dynamic connection can cause a slow down when starting, especially if the link connects to another object that may use a symbolic link itself.

Customer drivers (CBX) and ActiveX controls can cause issues.

[SCADADog]

Thanks for your reply.  I don't think there are any custom drivers or ActiveX controls being used, but I can re-check.

[wetback]

Was there any resolution on this? I've been having that same problem. Opening large trend files or opening two many processes at the same time. Anything that taxes the resources beyond some point will lock it up.

Thanks

[bovey_lee]

Hi, did you notice the cpu usage of lookout or citadel process when Lookout hung?

[wetback]

CPU, disk I/O, memory all look good, I have a couple of dumps, but cannot get anyone from NI to look at them. I wonder if Ryan Shi is still around. The application was written some time ago and doesn’t take advantage of mutable cores, but I don’t see the processor as being overloaded, I suspect it might be a thread priority problem, but I’m not an application programmer. We seem to be alone with this problem, so it might be something in the way we are using Lookout. In talking to people that have been using it long before I got here, they say it started about the time when they upgraded to Lookout 6.0, we are currently at 6.6, Windows 7, using the NIOPC server to communicate with mostly Siemens RTU’s and some Bristol. One issue seems to be that when the data base gets corrupted, I can delete it and Lookout will run better for a while. Usually Lookout needs to be rebooted every other day. We have five server locations that have to exchange information with each other, some of the links are not the best and I believe that is what’s causing the data corruption.

[bovey_lee]

Hi，I have couple questions to clarify:

1 The problem occurs when opening large trend files or opening too many processes, do you use hypertrend to show the data trend coming from NIOPC server? And do you log it into citadel database?  What's the What kind of processes do you mean?

2 What do you mean by lock-up? Lookout got frozen? Or the system frozen, you have to reboot? Do you get any warning?

3 You referred data base gets corrupted, is there any abnormal phenomenon about citadel process(nicitdl5.exe)? How do you know the database crrupted?

2 I'm from NI, would you like to send the dump files to me? I'll try my best help you to solve the problem.

[wetback]

1 The problem occurs when opening large trend files or opening too many processes, do you use hypertrend to show the data trend coming from NIOPC server?

Yes, we have one panel with 21 hypertrends on it, when it is opened, you cannot slide any windows around or do anything that would consume resources until all the trends are presented, if you do, you will get ‘Not Responding’, about an 80% chance it will never come back without a reboot.

And do you log it into citadel database?  Yes

What kind of processes do you mean?

Separate process files, client files that pertain to different areas of our operation, or locations, we have five different client process files that share information between them.

2 What do you mean by lock-up? Lookout got frozen? Or the system frozen, you have to reboot?

Lookout is frozen, I can still do things in windows outside of Lookout.

Do you get any warning?

No, the first indication is there are no trends, some operators report that the computer clock has stopped, data on the screen is old. Usually 10-20 minutes go by before the operators realize it is not working, then they reboot and everything comes back..

3 You referred data base gets corrupted, is there any abnormal phenomenon about citadel process(nicitdl5.exe)?

No, nothing I can see, all the processes are running.

 How do you know the database corrupted?

I cannot detach it, it just waits forever, if I delete it and reboot it creates another and will run fine until the next occurrence.

2 I'm from NI, would you like to send the dump files to me? I'll try my best help you to solve the problem.

I put it on your incoming server called ‘JP_lookout.6.6_dmp’

Thanks

[bovey_lee]

Hi，Can you upload the database with problem to FTP?

[wetback]

I don’t have one that is corrupted at this time. The database corruption is a theory, can anything be diagnosed from the dump that was taken when it was locked up, is something in a loop?