SCSM 2012 R2 > 2016 in-place upgrade

Leigh_Kilday

I've never been one to conduct an in-place upgrade on an enterprise application, however I wear many hats and the others are getting dusty. I'd like to make this procedure a lot faster and ideally not have to clone VMs or archive the old DW and reconfigure our custom cubes.

Besides being considered a generally bad idea, has anybody found any issues when upgrading SCSM to 2016 without migration?

My plan is to use the Cireson Lifecycle Management tool to make Test identical to Prod and see how the upgrade goes. If I don't need to roll back, I'll do the same with QA and then Production.

Tom_Hendricks

Yes, although it is not technically an upgrade issue.  The SCSM data warehouse jobs must all be in a state of "Not Started" for the upgrade to proceed (the DW is the first part of the upgrade), which means none of the batches can be stuck--all must be complete.

I have not seen an environment where absolutely 100% of the batches complete successfully, so getting to this state seems like wishful thinking.  Running DWMaintenance, forcing the jobs to run, deleting the HealthServiceState folder and restarting the DW Mgt services....none of that helps.  It also isn't necessary for normal operation of the DW, so none of this has mattered until upgrade time.

The TechNet article begins to frame a fix for this, then provides no detail, so it is not helpful. 

So in my case, I have unregistered and nuked the DW (which seems to be a popular "fix" in lieu of an actual solution), installed a fresh DW, then allowed it to re-populate.  None of my data are old enough to be lost in this process, so this works for me.  For others, it may be a non-starter.

Other than this, the upgrade works like a dream and exceeded my expectations.  Note that some of your install folders will still have "2012 R2" in the path after the upgrade, but I suspect this can be adjusted for those who are truly bothered by it.

Having said all this, I am still intrigued by the migration tool, particularly for how it might aid our SDLC.  Also, when moving to Windows Server 2016 as I would eventually like to, migration may be the only real choice.

Tom_Hendricks

Yes, although it is not technically an upgrade issue.  The SCSM data warehouse jobs must all be in a state of "Not Started" for the upgrade to proceed (the DW is the first part of the upgrade), which means none of the batches can be stuck--all must be complete.

I have not seen an environment where absolutely 100% of the batches complete successfully, so getting to this state seems like wishful thinking.  Running DWMaintenance, forcing the jobs to run, deleting the HealthServiceState folder and restarting the DW Mgt services....none of that helps.  It also isn't necessary for normal operation of the DW, so none of this has mattered until upgrade time.

The TechNet article begins to frame a fix for this, then provides no detail, so it is not helpful. 

So in my case, I have unregistered and nuked the DW (which seems to be a popular "fix" in lieu of an actual solution), installed a fresh DW, then allowed it to re-populate.  None of my data are old enough to be lost in this process, so this works for me.  For others, it may be a non-starter.

Other than this, the upgrade works like a dream and exceeded my expectations.  Note that some of your install folders will still have "2012 R2" in the path after the upgrade, but I suspect this can be adjusted for those who are truly bothered by it.

Having said all this, I am still intrigued by the migration tool, particularly for how it might aid our SDLC.  Also, when moving to Windows Server 2016 as I would eventually like to, migration may be the only real choice.

Leigh_Kilday

Thanks @Tom_Hendricks!

Jerry_Veldhuis

I'd like to hear how your in-place SCSM 2016 upgrade went.

We've noticed some rather troubling performance issues after the upgrade in our QA environment (which also includes SQL 2012 SP2 -> SP4). As a simple example, I have a powershell script that creates 1000 IRs which we ran before and after the upgrade. Which took 46min with SCSM 2012 and 64 min after the upgrade. 

We chose this test  https://docs.microsoft.com/en-us/system-center/scsm/whats-new-in-sm?view=sc-sm-2016 boasted "Time to create and update work items was greatly reduced using this improvement" and "Service Manger can more easily handle a large inflow of 45 work items per minute.", both of which we're no seeing.

Before you ask, yes, there is certainly some overhead in the powershell script where its pulling out a copy of the newly create WI to verify it, but the script did not change during the process, so this overhead should be about the same in both tests.

Has anyone else seen performance degradation going from scsm 2012 to 2016 ?

Tom_Hendricks

Yes, definitely, although I am not sure if it is a direct result of the newer version or not in most cases. The version of SQL Server you are running might matter, though.  Have you switched to SQL 2014+, too?  SCSM runs slower with the newer method of cardinality estimation.  You need to set it to use the 2012 method either globally or for the SCSM DB.  This is not the same as the DB Compatibility level (i.e. setting the compatibility to 2012 does not fix this).

We have had massive issues with performance (e.g. total work stoppages for several hours for certain time zones) due to undocumented workflows such as ECL Grooming (this is still needed even if you have disabled ECL), unapplied indexes, poor Exchange Connector performance, deadlocks, etc. but I honestly do not know if that is new for 2016 or just exacerbated by the larger number of tickets and config items we began to accumulate around the same time.  The specs on my DB Server look more like a virtual host than a guest now, and only after that has performance become "acceptable."

Jerry_Veldhuis

As part of our upgrade, we went from SQL 2012 SP2 to SQL 2012 SP4. I'm been googling, but can't find any solid references to SQL 2014 helping with SCSM 2016 performance.

I've love to hear your experience with unapplied indexes. We've applied index rebuilds based on > 30% fragmentation and reorg indexes if > 10%, but it hasn't improved the performance at all.

Tom_Hendricks

My point about SQL was the opposite--that newer versions of SQL Server perform worse for SCSM unless you set the cardinality estimation back to the 2012 method.

Applying the indexes made a very small improvement.  Getting rid of our Exchange Connectors made a huge difference.  Purging unnecessary data and getting the grooming jobs A.) working and B.) to stop running during business hours (2AM server time is business hours elsewhere in the world) made a much bigger difference.  Our grooming jobs were failing silently, leaving behind many tickets that were beyond the retention period and many other objects and entries in the ECL/History.