System Center Data Access Service Crashing after KB4498947 (Server 2016)

Adam_HillAdam_Hill Customer IT Monkey ✭
Not sure if this is a "Just me" or if this is more widespread but just wanted to throw the question/statement out there.

Right after applying this patch (May 2019 Service Stack Update), the Data Access service will not start/keeps crashing. Pretty much started right after the patch finished and persisted through reboots. This patch, being a stack update, isn't terribly easy to uninstall..likely going to need to roll the server back.

Anyone else experience this or have more data?

Answers

  • Brian_WiestBrian_Wiest Customer Super IT Monkey ✭✭✭✭✭
    I have SCSM 2016 UR7 running on Server 2016. After this months patches service is running without issue.
    I will note that this months release it took two reboots before the system finished applying the updates. I preformed a third reboot to make sure the services started in a clean fashion where they were not delayed by windows finishing other tasks. HTH
  • Emma_JenkinsEmma_Jenkins Customer IT Monkey ✭

    We are not having exactly the same issues but we applied KB4498947 and KB4494440 to our Dev and Test environments and we have some strange behaviour since which we are now investigating if there is something in there which is the cause.

    Our workflows did not recover themselves after the reboot following patching but all services were running. For us,  stopping the following services on the Management Server - Microsoft Monitoring Agent, System Centre Data Access Service and System Centre Management Configuration (in that order) and renaming the folder C:\Program Files\Microsoft System Center\Service Manager\Health Service State, then restarting the Services in the following order System Centre Management Configuration, System Centre Data Access Service, Microsoft Monitoring Agent helped clear the Service Manager cache and  get things working again. We don't have same issues but may be worth a shot before you roll back? Either way I do suspect something amiss with these updates.

  • Brian_WiestBrian_Wiest Customer Super IT Monkey ✭✭✭✭✭
    @Emma_Jenkins It is a common practice to have to perform those tasks on your MGT servers. To make sure we don't run into that issue we run a cache clear script every week to keep our farm healthy.

  • Emma_JenkinsEmma_Jenkins Customer IT Monkey ✭
    Hey @Brian_Wiest thanks for the tip on this. I am just giving it a go in our Dev environment but cannot get the 'deleting health service state items' to work. Any advice? Our file paths are slightly different to yours but i have amended them accordingly. My powershell is somewhat basic  :#  
  • Brian_WiestBrian_Wiest Customer Super IT Monkey ✭✭✭✭✭
    Update line 12 with the install path of SCSM. Wrote the script for SCSM 2012 but if you are running 2016 update as needed
    Example
    $SCSMHSS = "C:\Program Files (x86)\Microsoft System Center 2012 R2\Service Manager\Health Service State\*" #Local Path
    $SCSMHSS = "C:\Program Files\Microsoft System Center\Service Manager 2016\Health Service State\*" #Local Path


  • Emma_JenkinsEmma_Jenkins Customer IT Monkey ✭
    Update line 12 with the install path of SCSM. Wrote the script for SCSM 2012 but if you are running 2016 update as needed
    Example
    $SCSMHSS = "C:\Program Files (x86)\Microsoft System Center 2012 R2\Service Manager\Health Service State\*" #Local Path
    $SCSMHSS = "C:\Program Files\Microsoft System Center\Service Manager 2016\Health Service State\*" #Local Path


    Thanks @[email protected]_Wiest however we already amended that line, our file path is $SCSMHSS = "C:\Program Files\Microsoft System Center\Service Manager\Health Service State\*" #Local Path (doesn't have 2016 in the name but we are on 2016)
    At the end of the path it does they to resolve to a folder, doesn't seem to like the *


  • Brian_WiestBrian_Wiest Customer Super IT Monkey ✭✭✭✭✭
    The screenshot you have is correct as the script will delete all the folders listed.
    So either you have a permissions issue to that path or the parm is failing to pass properly.
    In line 48 replace the variable to the exact path and retest

    invoke-command -ComputerName $server -command { del "$SCSMHSS" -recurse }
    with
    invoke-command -ComputerName $server -command { del "C:\Program Files\Microsoft System Center\Service Manager 2016\Health Service State\*" -recurse }

    Make sure you are running PS as an Admin
  • Emma_JenkinsEmma_Jenkins Customer IT Monkey ✭
    edited May 21
    Apologies for hijacking @Adam_Hill o:)  

    I think this will be worth the effort here! Learning too  :)
  • Nick_FlintNick_Flint Customer Advanced IT Monkey ✭✭✭
    To keep the variable in place, I had to change

    invoke-command -ComputerName $server -command { del "$SCSMHSS" -recurse }

    to

    invoke-command -ComputerName $server -ScriptBlock { Remove-Item -Path $args[0] -force -recurse } -ArgumentList "$SCSMHSS"
  • Adam_HillAdam_Hill Customer IT Monkey ✭
    edited May 22
    No worries at all @Emma_Jenkins completely agreed!

    So I don't actually believe now that this patch was the core of my issue, but it was the patching process as a whole. Looking further into the event log, two things stuck out at me.

    During Patching, both of my management servers got the following error...

    Query notification processing failed due to a sql exception. 

    System.Data.SqlClient.SqlException (0x80131904): Cannot continue the execution because the session is in the kill state.
    A severe error occurred on the current command.  The results, if any, should be discarded.
       at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
       at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
       at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
       at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
       at System.Data.SqlClient.SqlDataReader.get_MetaData()
       at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
       at System.Data.SqlClient.SqlCommand.CompleteAsyncExecuteReader(Boolean isInternal, Boolean forDescribeParameterEncryption)
       at System.Data.SqlClient.SqlCommand.InternalEndExecuteReader(IAsyncResult asyncResult, String endMethod, Boolean isInternal)
       at System.Data.SqlClient.SqlCommand.EndExecuteReaderInternal(IAsyncResult asyncResult)
       at System.Data.SqlClient.SqlCommand.EndExecuteReader(IAsyncResult asyncResult)
       at Microsoft.EnterpriseManagement.DataAccessLayer.QueryNotificationManager.HandleNotifications(Object state)
    ClientConnectionId:78a76c27-c83e-4eac-889f-3b3deeda52ac
    Error Number:596,State:1,Class:21
    After this, I got a report saying that the main server was not processing e-mails, the exchange connector had ceased working.

    I've been having extensive problems with my Data Warehouse and have been working with MS directly to resolve them and happened to know about the Microsoft Health Monitoring Service. I restarted that on my primary server and everything began running again as expected (All of our connectors were stopped)

    However not so lucky on this secondary server that feeds the Cireson Analyst Portal. Now the repeated error I'm getting is a mixture of...

    The request to synchronize state for OpsMgr Health Service identified by "28bd3f1d-808f-e433-6755-7f46689a6a01" failed due to the following exception "System.Runtime.Remoting.RemotingException: Could not find Sdk Service uri named object. SDK Service may not be running.
    at Microsoft.Mom.ConfigService.Networking.ConnectionFactory.Connection.Sender.get_SdkServiceUri()
    at Microsoft.Mom.ConfigService.Networking.ConnectionFactory.Connection.Sender.GetSdkService()
    at Microsoft.Mom.ConfigService.Networking.ConnectionFactory.Connection.Sender.OnRetrieveSecureData(Guid healthServiceId, ReadOnlyCollection`1 addedSecureStorageReferences, ReadOnlyCollection`1 removedSecureStorageReferences, ReadOnlyCollection`1 modifiedSecureStorageReferences, ReadOnlyCollection`1 addedSecureStorageElements, ReadOnlyCollection`1 removedSecureStorageElements, ReadOnlyCollection`1 modifiedSecureStorageElements)
    at Microsoft.Mom.ConfigService.Networking.ConnectionFactory.Connection.OnRetrieveSecureData(Guid healthServiceId, ReadOnlyCollection`1 addedSecureStorageReferences, ReadOnlyCollection`1 removedSecureStorageReferences, ReadOnlyCollection`1 modifiedSecureStorageReferences, ReadOnlyCollection`1 addedSecureStorageElements, ReadOnlyCollection`1 removedSecureStorageElements, ReadOnlyCollection`1 modifiedSecureStorageElements)
    at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.CommunicationHelper.StateSyncRequestTask.ResponseWriter.SecureDataWriter.Write(StateSyncResponseMessage response)
    at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.CommunicationHelper.StateSyncRequestTask.ResponseWriter.WriteState(StateSyncResponseMessage response)
    at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.CommunicationHelper.StateSyncRequestTask.ResponseWriter.Write(StateSyncResponseMessage response)
    at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.CommunicationHelper.StateSyncRequestTask.ResponseWriter.Write(StateContext stateContext, ConfigurationItems configurationItems, StateSyncResponseMessage response, IConnection connection)
    at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.CommunicationHelper.StateSyncRequestTask.CreateResponse(Managers managers)
    at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.Managers.Synchronize(OnDoSynchronizedWork onDoSynchronizedWork)
    at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.CommunicationHelper.StateSyncRequestTask.Execute(Managers managers)
    at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.CommunicationHelper.StateSyncRequestTask.Run(Guid source, String cookie, Managers managers, IConfigurationDataAccessor dataAccessor, Stream stream, IConnection connection)".

    Followed by an information piece in the event log that keeps repeating

    OpsMgr has no configuration for management group <CorrectGroupName> and is requesting new configuration from the Configuration Service.

    And every time I start the Data Access Service, I get these two errors (And it crashes pretty immediately)

    Error 1
    An exception was thrown while initializing the service container.
    Exception message: Initialize
    Full exception: Feature of type 'Microsoft.EnterpriseManagement.ServiceDataLayer.ISecureStorageManagerFeature, Microsoft.EnterpriseManagement.DataAccessService.Core, Version=7.0.5000.0, Culture=neutral, PublicKeyToken=<CorrectToken>' cannot be added to the container.
    Error 2
    The System Center Data Access service failed due to an unhandled exception.  
    The service will attempt to restart. 
    Exception: 
    Microsoft.EnterpriseManagement.ConfigurationReaderException: Feature of type 'Microsoft.EnterpriseManagement.ServiceDataLayer.ISecureStorageManagerFeature, Microsoft.EnterpriseManagement.DataAccessService.Core, Version=7.0.5000.0, Culture=neutral, PublicKeyToken=<CorrectToken>' cannot be added to the container. ---> Microsoft.EnterpriseManagement.ComponentActivationException: The constructor for the component threw an exception. Please see the inner exception for more details. ---> System.Exception: Exception decrypting. Key not valid for use in specified state
    at Microsoft.EnterpriseManagement.Security.DPAPIWrapper.Decrypt(Byte[] cipherText)
    at Microsoft.EnterpriseManagement.Security.AsymmetricKeyManager.Initialize(Byte[] publicKey)
    at Microsoft.EnterpriseManagement.Security.SecureStorageManager.Initialize()
    at Microsoft.EnterpriseManagement.ServiceDataLayer.SecureStorageManagerFeatureImplementation..ctor()
    --- End of inner exception stack trace ---
    at Microsoft.EnterpriseManagement.ComponentActivator.Activate[T](ActivationContext`1 context)
    at Microsoft.EnterpriseManagement.SingletonLifetimeManager`1.GetComponent[K]()
    at Microsoft.EnterpriseManagement.FeatureContainer.GetFeatureInternal[T](Type type, String featureName)
    at Microsoft.EnterpriseManagement.FeatureContainer.AddFeatureInternal[T,V](ActivationContext`1 context, String featureName)
    --- End of inner exception stack trace ---
    at Microsoft.EnterpriseManagement.ConfigurationReaderHelper.ReadFeatures(XPathNavigator navi, IContainer container)
    at Microsoft.EnterpriseManagement.ConfigurationReaderHelper.Process()
    at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.Initialize(InProcEnterpriseManagementConnectionSettings configuration)
    at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.InitializeRunner(Object state)
    at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
    at System.Threading.ThreadHelper.ThreadStart(Object obj)
    I'm pretty stumped at this point, especially because the original SQL error was the same on both ends.
Sign In or Register to comment.