Moving the cachebuilder service and multiple portal servers

Scott_Mceleney

We are having some issues with our SCSM/Cireson environment and I wanted to run some options past the community to see what is possible, or if there are any options we have.

Current setup:

Workflow server, Data warehouse management server, cireson portal server (with cachebuilder), sql server and dw sql server.

We have built a second cireson portal server, which we intend to load balance with the other one, we hope this will mitigate some of our IIS issues, part of which is having too many connects on one server.

We have followed most guides on, tuning performance, and implemented some scripts to clear caches and clear the health state folder nightly.

Issues:

Occasionally we have issues with the cireson portal server, either with the cachebuilder service or the IIS service that force us to reboot the server. In this scenario we loose the portal, and when the server starts back up, the cachebuilder service starts again and does its thing - this takes out the portal for even longer for the analysts while it completes.

Occasionally we have an issue where something starts filling up the SQL tempdb - we are investigating what it is but whatever it is will take all the space we give it until we reboot the server. Has anyone ever seen an issue like this?

Options:

Cachebuilder service - should we move this to another server (the workflow server)? This would mean if we needed to reboot the portal server it would not be affected. Or should we leave it where it is but stop it from starting automatically and have it start in the middle of the night, should a reboot occur during the day? I know this would mean that for the rest of the day after the reboot, work items list views would not update but this would be better than our current problems of having it not working for analyst for some time.

We are also on the baseline version, would it be worth us moving to latest? also we are on SCSM 1801 would it be worth us upgrading to SCSM 2019, its a big piece of work and we want to justify the time we would have to spend on it. Is the risk worth the reward?

Thanks in advance

Shane_White

Hi @Scott_Mceleney

Some short points here to get the discussion flowing:

1. DO NOT, put cachebuilder on the workflow server, never add anything onto the workflow server as this server and it's resources take care of the heavy stuff and he workflows 😜
2. When these issus occur, do you get any errors in the web console or cache builder logs?
3. What version of baseline are you on?
4. In SQL, have you set your ServiceManagement DB Recovery Model to Simple rather than full?
5. The CacheBuilder Service will automatically do a full sync 24 hours from the last time it started, so if you restart it out of hours then it won't restart itself during hours.
6. Baseline vs. Latest depends on your organisation and their needs, please see this KB: https://kb.cireson.com/article/1243

Thanks,

Shane

Scott_Mceleney

@Shane_White Thanks for the response.

1. Okay noted, I think ill leave it on the first portal server.
2. Mostly we have issues where the connectors haven't completed, or something interrupts the cache builder so we have to restart the service. This is usually identified by some users being analysts and some not. The IIS issues stem from IIS taking up too many resources and flat lining the server respurces, usually a reboot sorts it otherwise it take a while to sort itself out, but then we get into the issues of the cachebuilder restarting when the server restarts. This is why i am thinking of setting it to start in the night rather than on first boot, so if we have to reboot for whatever reason, the use of the system is not hampered by the cachebuilder starting up (which takes about 40 minutes). Hopefully these issues will be lessened by having a second portal server.
3. We are on 9.4.1.2016 but looking to upgrade to the latest version (its currently on our test server)
4. SQL is indeed in simple recovery mode, but we still get a issues with the tempdb filling up. We have a SQL magician looking at it now
5. I think that will help if we start the cachebuilder in the night as per point 2.
6. Ill discuss with the team here about latest vs baseline.

We see a lot of connector issues. Is it worth upgrading to SCSM 2019 to try and help these issues?.

Cheers

Scott

Scott_Mceleney

@Shane_White or is it worth putting the cachebuilder on another server, any other server (other than workflow) that doesn't get accessed so if we do have to reboot the portal server it is protected from a reboot.

Scott_Mceleney

Also: this is the tempdb issue we get - https://social.technet.microsoft.com/Forums/ie/en-US/1d38dfd5-6f4c-47f0-bdaa-d8f3fb240011/large-temp-db-growth or something very similar. its the same process that is taking up all the space and we are following the suggested guidance but its not really helping.

Shane_White

@Scott_Mceleney

1. Whatever you do, moving the CacheBuilder service is not going to help your issue, if the cache builder is on another server and stops working, it is still going to affect the portal as that service is what is syncing the information from service manager, so please leave it on the same server as the portal.
2. a. Okay in regards to connectors, this is a Microsoft issue you need to investigate, but before you do we recommend that you reset the watermarks of the AD Connector and see if this makes a difference, if not then contact Microsoft and eliminate that as something causing issues. A link to a KB on how to do this is here: https://support.cireson.com/KnowledgeBase/View/2495/#/ b. What normally interrupts cache builder? Do you ever get any errors in the logs? c. Okay with IIS taking up resources, it could be the portal version, or it could be too much portal activity for one portal server to handle, so another portal server will definitely help this. Yes, definitely have CacheBuilder running at night, even if that means you use a script or task to do this to make sure.
3. I would definitely upgrade to the latest baseline version at least, as I believe this is the version on baseline with the improved cache builder and that should help significantly.
4. Okay that's good then, looking at the link you sent me, this is not something I have come across before but it could be possible, if it works please let me know and I will note this down for future!

Thanks,

Shane

Peter_Miklian

Just 2 fires at guess:

1. mentioning IIS taking too much resources - what about IIS pool recycling or something like that? Checking IIS logs?
2. SCSM log (Operations Manager) not showing any abnormalities?