NI SystemLink Service Manager Crashes

[judfm]

NI SystemLink Service Manager stops working after a while. In the crashlog_svcmgr, I see two things that stand out.

10/18/2023 09:10:44

NationalInstruments.Skyline.MessageBus.SkylineException: Attempt to publish on a closed channel: SvcMgrReportServicesResponseBroadcast

   at NationalInstruments.Skyline.MessageBus.MessagePublisher.PublishMessage(MessageHeader header, Byte[] body, Boolean mandatory)

   at NationalInstruments.Skyline.AkkaUtilities.SkylineGateway.PublishBroadcast(BroadcastMessage message)

   at NationalInstruments.Skyline.ServiceManager.Internal.ServiceStateTracker.MakeCurrentSnapshot(Boolean updateRemotes)

   at NationalInstruments.Skyline.ServiceManager.Internal.ServiceStateTracker.AddLocalServiceInstance(ServiceInstanceBase instance)

   at NationalInstruments.Skyline.ServiceManager.Internal.MessageManagerBase.SpawnerThreadWithUserData(Object state)

   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()

   at System.Threading.ThreadPoolWorkQueue.Dispatch()

=================================================

=================================================

WARNING: exception after exception handler already ran

10/18/2023 09:10:45

NationalInstruments.Skyline.MessageBus.SkylineException: Attempt to publish on a closed channel: TraceLoggerRegisterTracePointRoutedMessage

   at NationalInstruments.Skyline.MessageBus.MessagePublisher.PublishMessage(MessageHeader header, Byte[] body, Boolean mandatory)

   at NationalInstruments.Skyline.SkylineUtilities.TraceLogging.ProcessLogger.MakeOrLookupTracePoint(String name)

   at NationalInstruments.Skyline.ServiceManager.Internal.MessageManagerBase..ctor(String moduleName)

   at NationalInstruments.Skyline.ServiceManager.Internal.ServiceLauncher.CreateSkylineLauncher(ServiceLaunchable launchable, Boolean attachOnly)

   at NationalInstruments.Skyline.ServiceManager.Internal.MessageManagerBase.DelayedSpawnerThreadWithUserData(Object state)

   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()

   at System.Threading.ThreadPoolWorkQueue.Dispatch()

=================================================

=================================================

10/18/2023 13:09:21

NationalInstruments.Skyline.MessageBus.SkylineException: Attempt to publish on a closed channel: TraceLoggerRegisterTracePointRoutedMessage

   at NationalInstruments.Skyline.MessageBus.MessagePublisher.PublishMessage(MessageHeader header, Byte[] body, Boolean mandatory)

   at NationalInstruments.Skyline.SkylineUtilities.TraceLogging.ProcessLogger.MakeOrLookupTracePoint(String name)

   at NationalInstruments.Skyline.ServiceManager.Internal.MessageManagerBase..ctor(String moduleName)

   at NationalInstruments.Skyline.ServiceManager.Internal.ServiceLauncher.CreateSkylineLauncher(ServiceLaunchable launchable, Boolean attachOnly)

   at NationalInstruments.Skyline.ServiceManager.Internal.MessageManagerBase.DelayedSpawnerThreadWithUserData(Object state)

   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()

   at System.Threading.ThreadPoolWorkQueue.Dispatch()

=================================================

=================================================

WARNING: exception after exception handler already ran

10/18/2023 13:09:22

NationalInstruments.Skyline.MessageBus.SkylineException: Attempt to publish on a closed channel: SvcMgrHeartbeatBroadcast

   at NationalInstruments.Skyline.MessageBus.MessagePublisher.PublishMessage(MessageHeader header, Byte[] body, Boolean mandatory)

   at NationalInstruments.Skyline.AkkaUtilities.SkylineGateway.PublishBroadcast(BroadcastMessage message)

   at NationalInstruments.Skyline.ServiceManager.Internal.ServiceManagersTracker.SendHeartbeatAndCheckForMissedHeartbeats()

   at NationalInstruments.Skyline.ServiceManager.Internal.ServiceManagersTracker.HeartbeatThreadFunction()

   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

   at System.Threading.ThreadHelper.ThreadStart()

=================================================

[JoshuaP]

Have you checked to see if the system is periodically running out of memory? In addition, is this running on a dedicated server or a personal PC like a laptop? 

[JoshuaP]

Also, what version of SystemLink Server do you have installed?

[judfm]

Version 2022 Q1 installed on a Windows Server 2022 with 8 CPU & 20 GB RAM.

[judfm]

After a while, this is what happens and the service stops

[JoshuaP]

If you open the Tags application and look at the history for localhost.Health.Memory.UsePercentage what does it show? If your are periodically getting over 90% and if those spikes align with your crashes then that is the source of your issue and you should upgrade the amount of memory on your server or delete some data.

[judfm]

Thank you for your reply.

We have another server identical to this one, but the product version is different.

It's version 2023 Q3, but we're having problems connecting to the database from Diadem with this version, but we're not having any problems with the service stopping.

We've set up an identical server with version 2022 installed, and this time the connection from Diadem works, but the services stop after a delay.

Regarding verification from Tags, can you tell me how to access this feature?

[JoshuaP]

Should look something like this.

 

[judfm]

You will find the photo requested above