Slow "save" time

Service_Desk2

Hi, 

When we create a new request order or create a new incident/service request from the quick create the save time can be up to 60 seconds, and we sometime gets an prompt saying that the webpage stopped responding.
We do not see this behavior in the old sharepoint portal.

How can we troubleshoot this further? we have tried several different browsers and we have seen this behavior in several different version. today we are running 9.0.6.2016

Tom_Hendricks

The number of different potential causes for this is absolutely enormous.  There are many of threads about this that can be found in this forum already with a quick search, too.  However, here are a few suggestions to help narrow this down:

• Do you have a similarly long delay when saving from the SCSM console (from the same machine, just to make it a viable comparison)?
• Look at the event logs on your workflow server.  Are there entries about grooming job failures in there (as warnings, most likely)?
• Open SQL Server Mgt Studio and type the following command right after clicking save on the ticket: sp_who2.  Scroll down the list, maybe refresh it a few times until the save finishes.  Do you notice lots of blocking (some blocking is normal/expected, if it is very brief).  Follow the "blocked by" IDs until you find the one that everything else is blocked by.  Which queries are being used by the SPID that is blocking everything else?  (probably something that touches the Entity Change Log table, if I were to guess....)
• Have you ever (even once, maybe years ago) used the Cireson View Builder, and promoted a view to the portal?  If so, open PowerShell and with the SMLets loaded, type Get-ScsmView.  Do you see duplicates?  If so, remove the old ones.  Credit to @Adam_Dzyacky for that one!
• Do you have a lot of workflows?  Do they fail a lot or are they generally healthy?  Workflows are a Service Manager killer.  Using SMA or Orchestrator (or both) instead performs much better.  (They do still create a drag on performance, but a much more reasonable one).  Also, either retrying or ignoring failed (or stuck in "scheduled" indefinitely) workflows seems to help everything else run more smoothly.  There are a few good threads with scripts for that on this site.
  
• Do you have unapplied indexes on your ServiceManager or ServiceManagement databases?
• How many concurrent users do you have targeting your portal server?  You should technically never have more than 75 per server.  You can test this by opening a PowerShell session on (or targeting) the SCSM management server that your portal connects to (repeat on each if you load balance more than one).  With SMLets imported, type Get-ScsmConnectedUser.  You may see a much larger number than you expected, since you may have more user sessions than people (think multiple browser tabs, etc.).  You might need more web servers if you are too far over 150 or 75.
• Does IIS recycle your app pools more than once daily?  This suggestion won't work for everyone but if you have a web garden on your server, you can set your app pools to recycle after a physical memory limit is hit.  Having more than one app pool worker allows users to be transitioned from one to the other before it is recycled, avoiding the loss of their session, etc.  (There may be other/better methods for this too).  Remembering that you probably also have an SCSM management server running on your web server, you want to make sure that you keep roughly 20% of your RAM available after Windows, SCSM and IIS take their share.
• If you have made it through the rest of the bullet points without anything findings or needing any changes, take a look at your cache tables in the ServiceManagement DB for the Cireson Portal.  When was the last time you truncated those tables and rebuilt the cache?  The cache builder does not remove old records even if they have been groomed out of ServiceManager, despite what you might think.

Service_Desk2

Hi and thanks for the reply, i will answer them one by one.

• No, we don't have the same delay in the console on the portal server.
  But I have asked Helpdesk to use the console on the portal server the whole day.

• Can't find any errors/warnings about grooming failures.

• Regarding the sp_who2, is it blocks in the ServiceManager database i shall look? Performing sp_who2 gives me around 230 rows of data, so i need to filter that somehow to be able to see.

• The installation is about 3-4 months old, we have not created any views and promoted them to the portal.
  But running the cmdlet i can see that we have 1597 views, i guess most of them comes from imported SCOM MPs. No duplicates as i can see.

• We have around 120 active workflows, most of them are for notifications and assignment. No erors/warnings in the log regarding the workflows (except some notification warnings due to missing email)

• I will look into the indexes, need to ask DBA.

• Yesterday we had around 30 connections, right now we have 14.

• IIS recycles once per day.
  The server have around 70% of free memory.

• Is there any best practice for how often we should truncate those tables? and which tables are we talking if we specify?

Brian_Wiest

Just my thoughts on your above issue.

If I am reading into this correctly issue is specific to submission of data into the servicemanager database from the portal. Console is no issue and portal navigation is no issue. In your troubleshooting you have attempted a few different browsers all returning the same "save delay".

I would suggest to review your browser/workstation/server security controls. Firewall, antivirus, and GPO's

Sounds like the security program is grabbing the portal data submission package, scanning it, then allowing posting. 

Suggest to disable controls on the portal server to start with to test that theory.

HTH

Tom_Hendricks

Agreeing with @Brian_Wiest, especially after seeing your response, @Service_Desk2.   I have seen similar behavior in the portal, but I was also seeing it with the console and also had much more history in the Cireson tables too, hence the questions.  If you aren't seeing it in the console too and your install is this new, then I like Brian's approach.

Service_Desk2

Thanks for the replies. i haven't had the time to troubleshoot this any further but i must take some time now because more and more users is starting to complain.

i don't think there is any issues with the security settings since the old sharepoint portal worked with existing security settings.

EDIT: also, it is not slow every time. sometimes it goes really fast.

Brian_Wiest

If you have a request with only the affect user and assigned to user. Does it open faster then a request with multi related CI's?

Service_Desk2

to open work items, with our without CI´s have no notable difference for me.
the problem is to save new incidents, SR and to save new orders/request offerings.

i tried now this morning again.

I created an incident via the "+ new". i waited until the time 07:33:30 and then i hit "save".
it took time, i got a "page stopped responding"..
the finally at 07:34:15 (45s later) i get a POST in the IIS log.

i redid it at 07:49:30 and i had 55s delay between i hit save and i see a POST.

asked a colleague to try the same, at 08:05 she hit save and i have a POST at 08:05:04. so that took 4s. She is usual the one with alot of problems and often things work well for me.

i then created a new SR and it took 3s for me between i hit save and i see the POST.

i will attach a snippet from the webconsole.log from my first try for the day, maybe you can see something in it.

Service_Desk2

to open work items, with our without CI´s have no notable difference for me.
the problem is to save new incidents, SR and to save new orders/request offerings.

i tried now this morning again.

I created an incident via the "+ new". i waited until the time 07:33:30 and then i hit "save".
it took time, i got a "page stopped responding"..
the finally at 07:34:15 (45s later) i get a POST in the IIS log.

i redid it at 07:49:30 and i had 55s delay between i hit save and i see a POST.

asked a colleague to try the same, at 08:05 she hit save and i have a POST at 08:05:04. so that took 4s. She is usual the one with alot of problems and often things work well for me.

i then created a new SR and it took 3s for me between i hit save and i see the POST.

i have a snippet from the webconsole log from my first try. but i was not allowed to upload it.