What is causing Lookout to Lock-Up

[bovey_lee]

Hi, from the dump file, I could not detect any abnormal action, there is nothing in a loop, the code ends according to normal process, however, it finished at the point of creating time groups of native machine and remote machine and log data to the database, can you ensure the networks works well all the time? Hardwares works well? I mean workstation can acquire data from the hardware all the time? Have you tried to use fewer hypertrends? Such as three or four?  Have you used Lookout before? At that time, did this kind of problems occurs?

[wetback]

Hi, from the dump file, I could not detect any abnormal action, there is nothing in a loop, the code ends according to normal process, however, it finished at the point of creating time groups of native machine and remote machine and log data to the database, can you ensure the networks works well all the time?

No, the network has some real bottlenecks.

I’m trying to understand the details of “finished at the point of creating time groups of native machine and remote machine and log data to the database” did it close the files or was it still waiting on something to get to the next step? What happens if Lookout cannot access another machine in a given period of time? One of the things I suspected was the mapped drives timing out.

Hardware works well? I mean workstation can acquire data from the hardware all the time?

If you mean client workstation to server or server to RTU, some of our RTU’s are remote linked by radio and not too reliable, some server data is passed across the WAN to another server and we are at the mercy of TCP and carrier throughput, to get that data across.

Have you tried to use fewer hypertrends? Such as three or four?  

Yes less data improves the situation, anything that reduces the load on the program/cpu is better. As an example, while the trends are opening, if you slide a panel around the screen before it finishes painting the trends it will lock up.

Have you used Lookout before? At that time, did this kind of problems occurs?

My history with Lookout goes back four years, during that time it has always locked up, in talking to old timers it seems to have started with the upgrade to 6.0, but many other things happened during that time. We are currently at 6.6.

In searching the web and forum, this seems to be unique to us, this tread is the first situation that has similar problems. The person that set up Lookout is gone (he could have probably fixed it, given enough time) and others have touched it before me, so I keep looking for something that is implemented wrong.

I really appreciate you looking into this.

[bovey_lee]

Hello, generally speaking, when Lookout cannot access another machine in a given period of time, the connection will be broken, but the application can go on normally, there will be just a red fork on the display of the object connecting to remote resource. The program ends in the step of dowait() function, which is in the function of data logging, it should wait two seconds, however it freezes. Sometimes the dump file can show just part of information. You referred after you delete the database data, the performance will be better, so I guess the problem comes from local citadel cache. I have to say as citadel works, both lookout.exe and nicitdl5.exe will be executed,  can you create the dump files of both lookout.exe and nicitdl5.exe and send them to me , I will explore if there is some restrictions between these two processes. 

I think there's a high possibility that the problem comes from the application, because the memory and CPU are normal, so if you can simplify the application which still have the problem(for example with two computers, one retrieves 21 channels of data from another and displays it in hypertrend and log to database), it will be better. Do you mind sending me the process? After all, the best way to find out the root cause is to reproduce the problem, and also I can help you to see if there's some editing mistakes.

I have gone through all you replies, I still have some questions to clarify:

1 In your reply before, you said when the panel of 21 hypertrends is opened , about a 80% chance it will never come back, you also said when the database crashes, you will delete it, and usually you should reboot it every other day, did you mean the application can run normally two days and suddenly the database crash? So when will the problem exactly happens?  When you open the panel of hypertrends or during its running time? And how long is the interval between two problems?  Is it a fixed period of time? If not, how long is the longest time and the shortest time?

2 Can you monitor any difference of CPU or memory between normal and abnormal occasion? You referred they are OK, I wonder if there are some small changes. 

3 Because the problem is related to citadel, so when the problem occurs, can you launch NI MAX and choose Citadel 5 Universe browse to local computer to check for the properties of the trends and find if there  are abnormal cases. You can also send me the screen shot. Also try to check for the size of the database, I'm not sure if the database is full.

This kind of problem is hard to troubleshoot relatively, so I need your more information. Thanks.

[wetback]

I need to clarify something, realizing after I posted it, I said that after you open a panel with a bunch of trends on it, if you slide a panel around you will get “not responding”, that is on the client machine, that may or may not be related to the server hanging up.

To answer your questions; (I’m concentrating on the client side)

1 The hypertrend locking up issue on the client, I uploaded a dump of the client when it was perpetually locked up, I also uploaded the process file. This was on the developmental version on a windows 7, 64 bit machine, with nothing else running on it.

2 I watched CPU, disk, memory and network usage and found no anomalies. I can cause it to lock up 90% of the time if after it opens, clicking on the “Trends” tab at the top of the screen, this is the large trend file I mentioned.

3 I need to do more investigation there and get back to you, the data base was 662 Mbs. Uploaded a screenshot where it locked up while loading trends, with performance monitor in the view, note, it never recovered.

File is “JPLookoutIssue”

[wetback]

An addendum; those missing trends in the screenshot are all from remote stations on a slow radio link. How can that affect the performance of loading a trend when the data is coming from the Citadel database?

[bovey_lee]

Hi, there is a problem, I cannot open your sourse file using my Lookout 6.6, the warning window is as follows,

It seems that you use an object which is not registered or not included in Lookout, I don't know if you have used any other tool and I wonder if this will lead to the problem.

[wetback]

CTCALC.CBX uploaded, drop it in the Lookout folder.

Thanks

[bovey_lee]

From the dump file, there seems to be a thread deadlock. Could you send me the dump file of the nicitdl5.exe, I think that will be helpful.

[Dan786]

Have you got any luck with all this? A feedback would be great. Thanks

[wetback]

Still locking up .....

I made a discovery that I wanted to share.

I always thought that Lookout wasn’t getting data fast enough when it would lock up trying to paint large trends. But I found out that it is being overwhelmed with data, Lookout was written back in the days of 10Mb Ethernet. I found if you set the Ethernet port to 10 Mb full duplex, Lookout almost never locks up and you can open as many trends as you want. But the down side is current applications like email and excel spread sheets open very slowly. 100 Mb half duplex looks like it might be a good compromise. Ideally if somehow we could throttle the data down just to Lookout.