Lookout 6.2, Citadel Crash

[Mike@Forshock]

Ok, beating a dead horse... I think we are definitely getting to the end of our NI lifespan 😞

Background

System installed with Lookout 6.1 in 2008, ran well for about 1.5 years.  Random crashes, database corruption, etc. Initially believed to have been the machine (Software RAID 5 sucks).  New machine built in Oct. 2010; no RAID, Intel Core2 machine, XP SP3.  Upgraded to Lookout 6.2.  Could not import the 6.1 database, performance slowed to a crawl, believed to be from a corrupt database.  Created a new database.  

Today

Machine is experiencing system failures on Citadel and Lookout. We have nothing but AlarmDialer 4 (custom notification software), RealVNC Enterprise, Lookout 6.2, HP Printer and standard system related tools (TrueImage, AVG, Adobe Reader (Latest version, for manuals)) installed on this machine.  All updates installed.  

Latest Windows Mini-dumps: http://support.forshock.com/files/MRCWD/Minidump.zip

Latest (related) error messages:

---

Event Type: Error

Event Source: SQLBrowser
Event Category: None
Event ID: 1
Date: 12/21/2010
Time: 5:27:46 PM
User: N/A
Computer: SCADA
Description:
The SQL configuration for SQL is inaccessible or invalid.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.
Data:
0000: 4d 00 53 00 53 00 51 00   M.S.S.Q.
0008: 4c 00 53 00 65 00 72 00   L.S.e.r.
0010: 76 00 65 00 72 00 5c 00   v.e.r.\.
0018: 43 00 75 00 72 00 72 00   C.u.r.r.
0020: 65 00 6e 00 74 00 56 00   e.n.t.V.
0028: 65 00 72 00 73 00 69 00   e.r.s.i.
0030: 6f 00 6e 00 00 00         o.n...  
Event Type: ErrorEvent Source: SQLBrowserEvent Category: NoneEvent ID: 1Date: 12/21/2010Time: 5:27:46 PMUser: N/AComputer: SCADADescription:The SQL configuration for SQL is inaccessible or invalid.
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.Data:0000: 4d 00 53 00 53 00 51 00   M.S.S.Q.0008: 4c 00 53 00 65 00 72 00   L.S.e.r.0010: 76 00 65 00 72 00 5c 00   v.e.r.\.0018: 43 00 75 00 72 00 72 00   C.u.r.r.0020: 65 00 6e 00 74 00 56 00   e.n.t.V.0028: 65 00 72 00 73 00 69 00   e.r.s.i.0030: 6f 00 6e 00 00 00         o.n...  

---

Event Type: Information

Event Source: MSSQL$CITADEL

Event Category: (2)

Event ID: 26037

Date: 12/21/2010

Time: 10:01:26 AM

User: N/A

Computer: SCADA

Description:

The SQL Network Interface library could not register the Service Principal Name (SPN) for the SQL Server service. Error: 0x54b, state: 3. Failure to register an SPN may cause integrated authentication to fall back to NTLM instead of Kerberos. This is an informational message. Further action is only required if Kerberos authentication is required by authentication policies.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Data:

0000: b5 65 00 00 0a 00 00 00   µe......

0008: 0e 00 00 00 53 00 43 00   ....S.C.

0010: 41 00 44 00 41 00 5c 00   A.D.A.\.

0018: 43 00 49 00 54 00 41 00   C.I.T.A.

0020: 44 00 45 00 4c 00 00 00   D.E.L...

0028: 00 00 00 00               ....    

---

We get lots of these entries:

Event Type: Information

Event Source: MSSQL$CITADEL

Event Category: (2)

Event ID: 17401

Date: 12/21/2010

Time: 1:44:58 PM

User: N/A

Computer: SCADA

Description:

Server resumed execution after being idle 10079 seconds: user activity awakened the server. This is an informational message only. No user action is required.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Data:

0000: f9 43 00 00 0a 00 00 00   ùC......

0008: 0e 00 00 00 53 00 43 00   ....S.C.

0010: 41 00 44 00 41 00 5c 00   A.D.A.\.

0018: 43 00 49 00 54 00 41 00   C.I.T.A.

0020: 44 00 45 00 4c 00 00 00   D.E.L...

0028: 00 00 00 00               ....    

---

Event Type: Error

Event Source: Application Error

Event Category: None

Event ID: 1000

Date: 12/22/2010

Time: 7:29:18 AM

User: N/A

Computer: SCADA

Description:

Faulting application lookout.exe, version 6.2.0.49153, faulting module nicitmid.dll, version 5.6.0.49153, fault address 0x00069ffb.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Data:

0000: 41 70 70 6c 69 63 61 74   Applicat

0008: 69 6f 6e 20 46 61 69 6c   ion Fail

0010: 75 72 65 20 20 6c 6f 6f   ure  loo

0018: 6b 6f 75 74 2e 65 78 65   kout.exe

0020: 20 36 2e 32 2e 30 2e 34    6.2.0.4

0028: 39 31 35 33 20 69 6e 20   9153 in 

0030: 6e 69 63 69 74 6d 69 64   nicitmid

0038: 2e 64 6c 6c 20 35 2e 36   .dll 5.6

0040: 2e 30 2e 34 39 31 35 33   .0.49153

0048: 20 61 74 20 6f 66 66 73    at offs

0050: 65 74 20 30 30 30 36 39   et 00069

0058: 66 66 62 0d 0a            ffb..   

---

Event Type: Error

Event Source: System Error

Event Category: (102)

Event ID: 1003

Date: 12/21/2010

Time: 5:28:15 PM

User: N/A

Computer: SCADA

Description:

Error code 00000024, parameter1 001902fe, parameter2 b9cebbbc, parameter3 b9ceb8b8, parameter4 b965a667.

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Data:

0000: 53 79 73 74 65 6d 20 45   System E

0008: 72 72 6f 72 20 20 45 72   rror  Er

0010: 72 6f 72 20 63 6f 64 65   ror code

0018: 20 30 30 30 30 30 30 32    0000002

0020: 34 20 20 50 61 72 61 6d   4  Param

0028: 65 74 65 72 73 20 30 30   eters 00

0030: 31 39 30 32 66 65 2c 20   1902fe, 

0038: 62 39 63 65 62 62 62 63   b9cebbbc

0040: 2c 20 62 39 63 65 62 38   , b9ceb8

0048: 62 38 2c 20 62 39 36 35   b8, b965

0050: 61 36 36 37               a667    

[Mike@Forshock]

Just the answer I was looking for! 

Mike

[skibum019]

Mike,

You are not alone.  We've run Lookout for close to a decade and have had various problems with it over time.  We frequently have Citadel database crashes which take quite a bit of time to recover from.  We are on 6.1 currently, but have had the issue now for a few years.  I have not been able to identify any one thing that is causing the problem.  we have a second instance of Lookout at a separate plant that runs without incident, so to be honest, I'm at a bit of a loss as to what is causing the problem.  The only thing I can possibly point to in our situation is that the database(s) on the system that crashes are extremely large (up to 20GB) and have been upgrade from 4.x versions of Lookout over time. 

[Mike@Forshock]

Thanks for responding.  

We have decided to stop actively developing in Lookout.  We are currently transitioning to another SCADA/HMI package, developed by a company not too far from NI headquarters...

We have this happening in different impacts on multiple systems.  We have one system logging about 25 traces, very small process.  It exhibits the same problems.  

Have a good day,

Mike

[Mike@Forshock]

Year later.  We have a system with approximately 8.05GB in citadel.  System shutdown, restart.  Database is now taking over 1 hour to "rebuild" and is not complete.  Secondary has 7.99GB, same issue.

Processes will not function, system is sluggish, basically useless.

Long live Citadel... 

[Mike@Forshock]

Forgot to add:  In this time, the processes will not function!