Reduced outage time for SCSM Portal upgrades/updates

Leigh_Kilday

My team manages all enterprise applications, not just the ITSM toolset. I would like to reduce the time we spend physically present for out-of-hours patching.

I can see two ways this would work:

1. An unattended installation script that we could schedule for out-of-hours.
   The method we were using to extract licenses from the Cireson Licensing site no longer works for portal keys (I'm assuming the web call changed for the Environment Hash parameter). Either we'd need an updated method, or the portal gets a grace period of a week or so while we manually save out the keys from the site and transfer them to the DMZ.
2. Ability to patch portal servers during the day without affecting the live servers.
   I've been told the 7.4 portal will not run properly if the ServiceManagement DB has been upgraded to 8.1. I would like to be able to take half of the servers down and update them while users are still running on the other half, then patch the rest. The NLB handles this well without loss of data.

Tom_Hendricks

As for #1, I already run ours through a powershell script.  However, my script mainly repairs all the "damage" done by the installer after the fact (they fixed cert bindings, which was awesome, but it still also bulldozes session timers, caching, compression, virtual directory and app folder permissions, etc.  Not complaining because it is easily worked around, but I like to keep awareness high...).  When it triggers the installer (it calls the clickonce URL), then there has to be interaction from a user.

If we could get command-line switches or the equivalent for all the params that the installer asks for, then I could finally have this run as an unattended SMA job after hours.  I would even be OK with requiring a first-time manual install, and then just passing the account PWD in, utilizing the pre-filling of the fields that it currently does. 

Is that pretty much what you were thinking of, @Leigh_Kilday ?

Leigh_Kilday

Tom_Hendricks said:

As for #1, I already run ours through a powershell script.  However, my script mainly repairs all the "damage" done by the installer after the fact (they fixed cert bindings, which was awesome, but it still also bulldozes session timers, caching, compression, virtual directory and app folder permissions, etc.  Not complaining because it is easily worked around, but I like to keep awareness high...).  When it triggers the installer (it calls the clickonce URL), then there has to be interaction from a user.

If we could get command-line switches or the equivalent for all the params that the installer asks for, then I could finally have this run as an unattended SMA job after hours.  I would even be OK with requiring a first-time manual install, and then just passing the account PWD in, utilizing the pre-filling of the fields that it currently does. 

Is that pretty much what you were thinking of, @Leigh_Kilday ?

@Tom_Hendricks, yep that'd do the trick. I normally perform the "repair" tasks manually; would you please share your script? I want to pass a lot of this stuff on to a junior admin so anything I can do to help make that transition smoother would be greatly appreciated.

To elaborate further on my license issue I mentioned, keys are often invalid for upgrades and sometimes updates, so in our offline environment we need to get new keys at the time we download the latest portal version. There have been times where these keys have not been updated for the latest version and we only find this out when we've installed the portal. If we were to move to scheduled unattended installs, we'd need assurance that we have valid keys before we come in the next day and panic.

Leigh_Kilday

I've built a solution using TFS. I can't get my scripts out of the network, but I can explain what I've done because it's not that complicated, but it does assume you know a bit about build and release in TFS/VSTS.

At this time the solution does not cater for new installations.

Two+ files are required:

config_{env}.psd1

1. Hash table of environment-specific values, such as DB server, app name etc. This needs to contain everything not common between environments.
2. There will be as many of these files as there are environments.
3. Selected by changing the deployEnvironment variable (explained below).

deploy.ps1

1. Contains a hash table of the global parameters required to run InstallPortal.ps1 (provided within the source files).
2. Reads from the config psd1 files for environment-specific values.
3. If the current server is running the CacheBuilder, set a flag to run the SQL jobs below. Otherwise just install the portal. This is why it can't be run for a fresh installation.
4. Invoke-SQLcmd executes a custom SQL Server Agent job to remove the ServiceManagement and CiresonAnalytics databases from the AAG.
5. InstallPortal.ps1 run with the parameters defined by the hash table.
6. Invoke-SQLcmd executes another Agent job to add the databases back into the AAG and set the failover mode to automatic.

How it works

1. Put source files in a specific folder on each server (can be scripted). My script expects that there is only one version of the source files in there at a time.
2. Before running a Build in TFS/VSTS, manually check the InstallPortal.ps1 parameters against the previous version. Update files if necessary.
3. The TFS Build definition publishes the files as an artifact. No need to create a new build if the parameters have not changed.
4. The Release definition copies the artifact to the target servers (defined in a variable group) and runs the deploy.ps1 script with the following variables as arguments; deployEnvironment (the only environment-level variable), appPoolUser, appPoolPassword, cacheUser, cachePassword, analyticsUser, analyticsPassword. You could put all of the config psd1 variables in here, but I found it neater to use the hash tables and just use TFS to store my creds securely.

Whatever account you run the deploy.ps1 as requires temporary sysadmin on the SQL instance, admin permissions in SCSM, and read/write access to the CiresonPortal folder on the filesystem. My DBAs were trying to avoid elevating the account to sysadmin, and we got really close to defining specific permissions but there are some scripts invoked by the provided installation script that require it.

The advantages of using TFS to release the updates include:

1. For me the release is completed in under three minutes. The deploy.ps1 script can be run concurrently so it's no slower.
2. If the parameters in the InstallPortal.ps1 script are the same, updating the portal is as simple as placing the new files into the right place and creating a new release.
3. Approval gates between environments allows for testing and verification. Once completed, the release can be resumed.
4. It's clear which environments have been updated.
5. All information relating to the release is in a single place.
6. Deployment to environments can be scheduled. For example, I can schedule the release to occur in Prod at 23:00, in accordance with my Change Request, and I don't need to be here.

Leigh_Kilday

I forgot to mention you, @Tom_Hendricks. You may be interested in my last.

Tom_Hendricks

I find the whole setup to be interesting, actually.  Thanks for sharing the outline.  Bookmarked!