NI SystemLink Server Manger Crashes

[ryan_sheppard]

NI SystemLink Service Manager crashes after a minute or two of restarting. In the crashlog_svcmgr I see two things that stand out.

"System.MissingFieldException: Configuration file not found for TraceLogger
at NationalInstruments.Skyline.ServiceManager.Internal.RuntimeDescriptorFileProcessor.AddImplicitDependencies(List`1 files)
at NationalInstruments.Skyline.ServiceManager.Internal.RuntimeDescriptorFileProcessor.ProcessRuntimeDescriptorFiles(List`1& newFiles, List`1& changedFiles, List`1& deletedFiles)
at NationalInstruments.Skyline.ServiceManager.Internal.ServiceManager.ProcessDescriptorFilesIfNeeded()
at NationalInstruments.Skyline.ServiceManager.Internal.MessageManagerBase.SpawnerThread(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()

NationalInstruments.Skyline.MessageBus.SkylineException: Could not connect to RabbitMQ broker ---> RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> RabbitMQ.Client.Exceptions.ConnectFailureException: Connection failed ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:5673
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at System.Net.Sockets.Socket.Connect(IPAddress address, Int32 port)
at RabbitMQ.Client.TcpClientAdapter.<ConnectAsync>d__2.MoveNext()

The error from my system event viewer: The NI SystemLink Service Manager service terminated with the following error:
%%3762504530

I have a clean install of SystemLink on my server. I have stopped the service, deleted the __CACHE__ file, restarted the service and still have the same issue. 

[ryan_sheppard]

Also found in the Application Viewer:

Application: NationalInstruments.ServiceManager.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.MissingFieldException
at NationalInstruments.Skyline.ServiceManager.Internal.RuntimeDescriptorFileProcessor.AddImplicitDependencies(System.Collections.Generic.List`1<NationalInstruments.Skyline.ServiceManager.DataObjects.ServiceDescriptorFile>)
at NationalInstruments.Skyline.ServiceManager.Internal.RuntimeDescriptorFileProcessor.ProcessRuntimeDescriptorFiles(System.Collections.Generic.List`1<NationalInstruments.Skyline.ServiceManager.DataObjects.ServiceDescriptorFile> ByRef, System.Collections.Generic.List`1<NationalInstruments.Skyline.ServiceManager.DataObjects.ServiceDescriptorFile> ByRef, System.Collections.Generic.List`1<NationalInstruments.Skyline.ServiceManager.DataObjects.ServiceDescriptorFile> ByRef)
at NationalInstruments.Skyline.ServiceManager.Internal.ServiceManager.ProcessDescriptorFilesIfNeeded()
at NationalInstruments.Skyline.ServiceManager.Internal.MessageManagerBase.SpawnerThread(System.Object)
at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
at System.Threading.ThreadPoolWorkQueue.Dispatch()

Also:

Faulting application name: NationalInstruments.ServiceManager.exe, version: 19.0.0.51204, time stamp: 0x5c9e528c
Faulting module name: KERNELBASE.dll, version: 10.0.14393.2999, time stamp: 0x5ce36c36
Exception code: 0xe0434352
Fault offset: 0x0000000000034c48
Faulting process id: 0x22b0
Faulting application start time: 0x01d53b3572686cbc
Faulting application path: C:\Program Files\National Instruments\Shared\Skyline\NationalInstruments.ServiceManager.exe
Faulting module path: C:\Windows\System32\KERNELBASE.dll
Report Id: b23f57e9-bece-4faf-9a8e-748ea1d80ff6
Faulting package full name:
Faulting package-relative application ID: 

[Lenard_50]

Looks like it's been over a year with no update.  I have the exact same problem under your "Also: report" with nationalinstruments.servicemanager.exe and kernelbase.dll.  

Have you heard or figured out a solution?

[JoshuaP]

If your problem is exactly the same as above it maybe an indication that the TraceLogger file is missing or invalid.  What are the contents of the following file on your server.

C:\ProgramData\National Instruments\Skyline\Install\Services\Descriptors\TraceLogger.json

[Lenard_50]

I have the TraceLogger.json file: 

{
"ServiceDescriptor": {
"Name": "TraceLogger",
"Version": "1.0.0",
"ServiceGroups": [
{
"Singleton": false,
"DefaultInstanceCount": 1,
"StartMode": "Auto",
"Properties": {
"UserAccountName": "*S-1-5-##"
}
}
]
}
}

## is a number between 01-99.

[JoshuaP]

That looks correct to me.  Is your server working at all or always failing in this way?  You may have a corrupt installation and if that is the case, I would suggest doing a repair or upgrade to the latest version if you aren't already running 2020 R3.

[Lenard_50]

The server is working 99% of the time. 

It may be the installation.  That may be in the future.  Unfortunately the system needs to stay at it's current version 2019SP1.  Additionally I've found an issue with virus updates that may be related.  

Thanks for the help.

[JoshuaP]

If it is working most of the time and only periodically failing it could be the result of running the server low on memory.  Windows doesn't handle running out of memory very well and you can end up with services crashing in bizarre ways.