SCSM console freezes and won't open again after editing offerings
Ingrid_Glatz
Customer Adept IT Monkey ✭✭
Since quite a while, we have an issue that a lot of people report a non-responsive SCSM console or even the portal that won't list users to assign them to a ticket or they cannot save a ticket or others. SCSM is 2012 R2 UR9 on WinSrv 2012 R2, SQL 2014 SP1.
Keeping an eye on it to figure out the reason, it looks like that editing offerings seem to cause some sort of overload on the SQL server which causes the consoles and web portal not to be responsive any longer. I'm the only person that works on offerings and I usually use a Terminalserver with the SCSM console installed to create new offerings and modify existing ones. Not always, but sometimes, when saving an offering form, the console freezes. I have to kill the console task and the console won't start again, staying in "Initializing". This situation will stay for about 10-15 minutes, then the console will open up again, while SQL CPU time seems to decrease. During this period, a lot of analysts, working either with the console on the Terminalserver, their Windows 10 machine with the local console or the Cireson portal report that they cannot continue their work because of the unresponsive console or web site. The console cannot be opened anywhere, neither on the Terminalserver nor the SCSM Managementserver nor the Windows 10 machine.
It doesn't happen always when editing offerings, but I'm actually afraid right now to work on offerings in general. I was told to only work on the primary Managementserver when editing offerings from now on to figure out whether this issue persists or if it's an issue when working on the Terminalserver.
Does anybody experience the same or similar issue and has figured out how to solve it? Somebody said that a customer of him had the same because of the Cireson products when working with a local console, saving offerings and had a freeze of 10 minutes. He didn't mention whether it might be related to the portal, the Asset management or another product.
Tipps are more than welcome, I definitely cannot skip working on offerings. Thanks.
Ingrid
Best Answer
-
Brian_Wiest
Customer Super IT Monkey ✭✭✭✭✭
Based on this and other posts you have some tuning to perform on your farm and sql farm. We engaged MS for a RAP (https://services.premier.microsoft.com/assess) found a number of performance issues that makes all the difference.
When you are making the save event on the management pack, if you watch the server task manager you will see the Microsoft.Mom.Sdk.ServiceHost.exe process peg the CPU. (And afterwards on the Portal server the IIS Worker Process peg the CPU)
In working with Cireson and Microsoft did not get to any solution to remove the CPU spike it ended up just being what it is.
Now in my testing and opinion it has to do with the health state service coping the management pack to the different end points, and the larger the management pack the longer it takes.
Now I have a pretty sizable farm and any change I make to the MP's it spikes the cpu for a minute or so. typically no one picks up on it if I work slowly on my edits. (pausing in between edits for the system to catch up) I did once re-import a large MP and it spiked the processors for 5 minutes.
Also in experience the more people using the system the longer it takes, so I try to make my changes first thing in the morning or late in the afternoon, this cuts down on the CPU spike time.
HTH
9
Answers
When you are making the save event on the management pack, if you watch the server task manager you will see the Microsoft.Mom.Sdk.ServiceHost.exe process peg the CPU. (And afterwards on the Portal server the IIS Worker Process peg the CPU)
In working with Cireson and Microsoft did not get to any solution to remove the CPU spike it ended up just being what it is.
Now in my testing and opinion it has to do with the health state service coping the management pack to the different end points, and the larger the management pack the longer it takes.
Now I have a pretty sizable farm and any change I make to the MP's it spikes the cpu for a minute or so. typically no one picks up on it if I work slowly on my edits. (pausing in between edits for the system to catch up) I did once re-import a large MP and it spiked the processors for 5 minutes.
Also in experience the more people using the system the longer it takes, so I try to make my changes first thing in the morning or late in the afternoon, this cuts down on the CPU spike time.
HTH
Hi Brian,
thanks. This is not what I've expected to read but it explains pretty good the cause of the issue and sounds comprehensible. I would have liked to have a better solution. I can definitely confirm that the ServiceHost.exe process is also quite busy during this freeze.
We already checked proper installation and performance of the SQL server, but maybe have to dig a little bit deeper. I'll ask whether I can do a RAP with MS, this might shed some more light into the issue. I don't know whether an MP is considered being large if its XML has 3.7MB. That was the largest one I edited last week where the freeze happened afterwards.
Unfortunately, I cannot do all the changes during off-peak hours. Early morning wouldn't be a problem, but I'm mostly off before 4pm. Sometimes, changes have to be done more quickly.
Ingrid
Primary workflow server 2 proc 4 core (8 core total) 32GB ram
Portal server 4 proc 4 core (16 core total) 48GB ram
This use case is all about the performance of your hosts. Not a SQL issue.
Note we also clear our SCSM cache weekly to help the health state service.
Repeating what @Brian_Wiest said, just because of how important it is, spacing out your edits (giving them some time to fully process) helps, whereas clicking the "OK" button on all these dialogs many times a minute can overwhelm the management servers.