System Center Data Access Service Crashing after KB4498947 (Server 2016)

Adam_Hill

Not sure if this is a "Just me" or if this is more widespread but just wanted to throw the question/statement out there.

Right after applying this patch (May 2019 Service Stack Update), the Data Access service will not start/keeps crashing. Pretty much started right after the patch finished and persisted through reboots. This patch, being a stack update, isn't terribly easy to uninstall..likely going to need to roll the server back.

Anyone else experience this or have more data?

Brian_Wiest

I have SCSM 2016 UR7 running on Server 2016. After this months patches service is running without issue.
I will note that this months release it took two reboots before the system finished applying the updates. I preformed a third reboot to make sure the services started in a clean fashion where they were not delayed by windows finishing other tasks. HTH

Emma_Jenkins

We are not having exactly the same issues but we applied KB4498947 and KB4494440 to our Dev and Test environments and we have some strange behaviour since which we are now investigating if there is something in there which is the cause.

Our workflows did not recover themselves after the reboot following patching but all services were running. For us,  stopping the following services on the Management Server - Microsoft Monitoring Agent, System Centre Data Access Service and System Centre Management Configuration (in that order) and renaming the folder C:\Program Files\Microsoft System Center\Service Manager\Health Service State, then restarting the Services in the following order System Centre Management Configuration, System Centre Data Access Service, Microsoft Monitoring Agent helped clear the Service Manager cache and  get things working again. We don't have same issues but may be worth a shot before you roll back? Either way I do suspect something amiss with these updates.

Brian_Wiest

@Emma_Jenkins It is a common practice to have to perform those tasks on your MGT servers. To make sure we don't run into that issue we run a cache clear script every week to keep our farm healthy.

You can see more about it here https://community.cireson.com/discussion/3905/full-scsm-and-cireson-cache-flush-powershell-script

Emma_Jenkins

Hey @Brian_Wiest thanks for the tip on this. I am just giving it a go in our Dev environment but cannot get the 'deleting health service state items' to work. Any advice? Our file paths are slightly different to yours but i have amended them accordingly. My powershell is somewhat basic   

Brian_Wiest

Update line 12 with the install path of SCSM. Wrote the script for SCSM 2012 but if you are running 2016 update as needed

Example

$SCSMHSS = "C:\Program Files (x86)\Microsoft System Center 2012 R2\Service Manager\Health Service State\*" #Local Path

$SCSMHSS = "C:\Program Files\Microsoft System Center\Service Manager 2016\Health Service State\*" #Local Path

Emma_Jenkins

Brian_Wiest said:

Update line 12 with the install path of SCSM. Wrote the script for SCSM 2012 but if you are running 2016 update as needed

Example

$SCSMHSS = "C:\Program Files (x86)\Microsoft System Center 2012 R2\Service Manager\Health Service State\*" #Local Path

$SCSMHSS = "C:\Program Files\Microsoft System Center\Service Manager 2016\Health Service State\*" #Local Path

Thanks @Br@Brian_Wiest however we already amended that line, our file path is $SCSMHSS = "C:\Program Files\Microsoft System Center\Service Manager\Health Service State\*" #Local Path (doesn't have 2016 in the name but we are on 2016)
At the end of the path it does they to resolve to a folder, doesn't seem to like the *

Brian_Wiest

The screenshot you have is correct as the script will delete all the folders listed.

So either you have a permissions issue to that path or the parm is failing to pass properly.

In line 48 replace the variable to the exact path and retest

invoke-command -ComputerName $server -command { del "$SCSMHSS" -recurse }

with

invoke-command -ComputerName $server -command { del "C:\Program Files\Microsoft System Center\Service Manager 2016\Health Service State\*" -recurse }

Make sure you are running PS as an Admin

Emma_Jenkins

Apologies for hijacking @Adam_Hill   

I think this will be worth the effort here! Learning too 

Nick_Flint

To keep the variable in place, I had to change

invoke-command -ComputerName $server -command { del "$SCSMHSS" -recurse }

to

invoke-command -ComputerName $server -ScriptBlock { Remove-Item -Path $args[0] -force -recurse } -ArgumentList "$SCSMHSS"

Adam_Hill

No worries at all @Emma_Jenkins completely agreed!

So I don't actually believe now that this patch was the core of my issue, but it was the patching process as a whole. Looking further into the event log, two things stuck out at me.

During Patching, both of my management servers got the following error...

Query notification processing failed due to a sql exception. 

System.Data.SqlClient.SqlException (0x80131904): Cannot continue the execution because the session is in the kill state.
A severe error occurred on the current command.  The results, if any, should be discarded.

   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)

   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)

   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)

   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()

   at System.Data.SqlClient.SqlDataReader.get_MetaData()

   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)

   at System.Data.SqlClient.SqlCommand.CompleteAsyncExecuteReader(Boolean isInternal, Boolean forDescribeParameterEncryption)

   at System.Data.SqlClient.SqlCommand.InternalEndExecuteReader(IAsyncResult asyncResult, String endMethod, Boolean isInternal)

   at System.Data.SqlClient.SqlCommand.EndExecuteReaderInternal(IAsyncResult asyncResult)

   at System.Data.SqlClient.SqlCommand.EndExecuteReader(IAsyncResult asyncResult)

   at Microsoft.EnterpriseManagement.DataAccessLayer.QueryNotificationManager.HandleNotifications(Object state)

ClientConnectionId:78a76c27-c83e-4eac-889f-3b3deeda52ac

Error Number:596,State:1,Class:21

After this, I got a report saying that the main server was not processing e-mails, the exchange connector had ceased working.

I've been having extensive problems with my Data Warehouse and have been working with MS directly to resolve them and happened to know about the Microsoft Health Monitoring Service. I restarted that on my primary server and everything began running again as expected (All of our connectors were stopped)

However not so lucky on this secondary server that feeds the Cireson Analyst Portal. Now the repeated error I'm getting is a mixture of...

The request to synchronize state for OpsMgr Health Service identified by "28bd3f1d-808f-e433-6755-7f46689a6a01" failed due to the following exception "System.Runtime.Remoting.RemotingException: Could not find Sdk Service uri named object. SDK Service may not be running.
at Microsoft.Mom.ConfigService.Networking.ConnectionFactory.Connection.Sender.get_SdkServiceUri()
at Microsoft.Mom.ConfigService.Networking.ConnectionFactory.Connection.Sender.GetSdkService()
at Microsoft.Mom.ConfigService.Networking.ConnectionFactory.Connection.Sender.OnRetrieveSecureData(Guid healthServiceId, ReadOnlyCollection`1 addedSecureStorageReferences, ReadOnlyCollection`1 removedSecureStorageReferences, ReadOnlyCollection`1 modifiedSecureStorageReferences, ReadOnlyCollection`1 addedSecureStorageElements, ReadOnlyCollection`1 removedSecureStorageElements, ReadOnlyCollection`1 modifiedSecureStorageElements)
at Microsoft.Mom.ConfigService.Networking.ConnectionFactory.Connection.OnRetrieveSecureData(Guid healthServiceId, ReadOnlyCollection`1 addedSecureStorageReferences, ReadOnlyCollection`1 removedSecureStorageReferences, ReadOnlyCollection`1 modifiedSecureStorageReferences, ReadOnlyCollection`1 addedSecureStorageElements, ReadOnlyCollection`1 removedSecureStorageElements, ReadOnlyCollection`1 modifiedSecureStorageElements)
at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.CommunicationHelper.StateSyncRequestTask.ResponseWriter.SecureDataWriter.Write(StateSyncResponseMessage response)
at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.CommunicationHelper.StateSyncRequestTask.ResponseWriter.WriteState(StateSyncResponseMessage response)
at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.CommunicationHelper.StateSyncRequestTask.ResponseWriter.Write(StateSyncResponseMessage response)
at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.CommunicationHelper.StateSyncRequestTask.ResponseWriter.Write(StateContext stateContext, ConfigurationItems configurationItems, StateSyncResponseMessage response, IConnection connection)
at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.CommunicationHelper.StateSyncRequestTask.CreateResponse(Managers managers)
at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.Managers.Synchronize(OnDoSynchronizedWork onDoSynchronizedWork)
at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.CommunicationHelper.StateSyncRequestTask.Execute(Managers managers)
at Microsoft.Mom.ConfigService.Engine.ConfigurationEngine.CommunicationHelper.StateSyncRequestTask.Run(Guid source, String cookie, Managers managers, IConfigurationDataAccessor dataAccessor, Stream stream, IConnection connection)".

Followed by an information piece in the event log that keeps repeating

OpsMgr has no configuration for management group <CorrectGroupName> and is requesting new configuration from the Configuration Service.

And every time I start the Data Access Service, I get these two errors (And it crashes pretty immediately)

Error 1

An exception was thrown while initializing the service container.
Exception message: Initialize
Full exception: Feature of type 'Microsoft.EnterpriseManagement.ServiceDataLayer.ISecureStorageManagerFeature, Microsoft.EnterpriseManagement.DataAccessService.Core, Version=7.0.5000.0, Culture=neutral, PublicKeyToken=<CorrectToken>' cannot be added to the container.

Error 2

The System Center Data Access service failed due to an unhandled exception.  
The service will attempt to restart. 
Exception: 
Microsoft.EnterpriseManagement.ConfigurationReaderException: Feature of type 'Microsoft.EnterpriseManagement.ServiceDataLayer.ISecureStorageManagerFeature, Microsoft.EnterpriseManagement.DataAccessService.Core, Version=7.0.5000.0, Culture=neutral, PublicKeyToken=<CorrectToken>' cannot be added to the container. ---> Microsoft.EnterpriseManagement.ComponentActivationException: The constructor for the component threw an exception. Please see the inner exception for more details. ---> System.Exception: Exception decrypting. Key not valid for use in specified state
at Microsoft.EnterpriseManagement.Security.DPAPIWrapper.Decrypt(Byte[] cipherText)
at Microsoft.EnterpriseManagement.Security.AsymmetricKeyManager.Initialize(Byte[] publicKey)
at Microsoft.EnterpriseManagement.Security.SecureStorageManager.Initialize()
at Microsoft.EnterpriseManagement.ServiceDataLayer.SecureStorageManagerFeatureImplementation..ctor()
--- End of inner exception stack trace ---
at Microsoft.EnterpriseManagement.ComponentActivator.Activate[T](ActivationContext`1 context)
at Microsoft.EnterpriseManagement.SingletonLifetimeManager`1.GetComponent[K]()
at Microsoft.EnterpriseManagement.FeatureContainer.GetFeatureInternal[T](Type type, String featureName)
at Microsoft.EnterpriseManagement.FeatureContainer.AddFeatureInternal[T,V](ActivationContext`1 context, String featureName)
--- End of inner exception stack trace ---
at Microsoft.EnterpriseManagement.ConfigurationReaderHelper.ReadFeatures(XPathNavigator navi, IContainer container)
at Microsoft.EnterpriseManagement.ConfigurationReaderHelper.Process()
at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.Initialize(InProcEnterpriseManagementConnectionSettings configuration)
at Microsoft.EnterpriseManagement.ServiceDataLayer.DispatcherService.InitializeRunner(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart(Object obj)

I'm pretty stumped at this point, especially because the original SQL error was the same on both ends.