Anyone else seeing 100% CPU usage on web servers with Cireson Portal version 9.1.2?

Jason_Meyer

Anyone else seeing 100% CPU usage with Portal version 9.1.2?   We upgraded last night and roughly 18 hours later both our web front end servers are pegged at 100% CPU.   We are preparing to roll back to the previous version.

Both servers have been rebooted and have returned to 100% CPU.   The website is working but very slow.

Any triage or troubleshooting that we can do?

Brian_Wiest

What service is eating up the CPU?

Jason_Meyer

IIS.

Shane_White

@Jason_Meyer

How many analysts/end users do you have using the Portal? It could be something to do with the sessions not ending.

In session state in IIS what time have you got it set to?

Thanks,
Shane.

Gerhard_Goossens

I also had this issue on my test VM, downgraded back to 9.1.0

Brian_Wiest

After downgrading did you perform an iisreset /restart?

I am running v9.2.1.2016 on Server 2016 and not seeing any performance issues.

Gerhard_Goossens

@Brian_Wiest
Nope, just reinstalled the old version and everything was back to normal.
No IIS/CacheBuilder or VM bounce

Still wanted to investigate but had to rollback for other testing.

Jason_Meyer

@Shane_White - We have about 15k users, 1k analysts.  It was running 9.1.1 with no issues for 4-5 months..  Day after the upgrade to 9.1.2, after about 2 hours of load it went to 100% CPU.  Asking our engineers about the session state time question.

We rolled back as well and are monitoring the system today.

Shane_White

@Jason_Meyer

I would advise putting in a Support Ticket for this but in the meantime it may be a performance issue with big environments.

@Brian_Wiest / @Gerhard_Goossens
how many users and analysts do you have?

Thanks,
Shane.

Brian_Wiest

In production about 1K analysts and ~25K users. But for production we are running 5 Portal servers. 1 is just for the cache builder and the other 4 are connected to the NLB.

Currently in the UAT we have only about 10-100 testing depending on what is getting UAT approval to move to production.

Currently we are at 0.1% on IIS Worker process.

Shane_White

@Jason_Meyer

How many servers are you running? When you upgraded, did you check for any errors in the logs? Or in event viewer?

Thanks,
Shane.

Gerhard_Goossens

Shane_White said:

@Jason_Meyer

I would advise putting in a Support Ticket for this but in the meantime it may be a performance issue with big environments.

@Brian_Wiest / @Gerhard_Goossens
how many users and analysts do you have?

Thanks,
Shane.

We have 60k users and about 150 analysts. I would guess that about 1000 of the users use the system on a monthly basis.

For now, the system is only used for IT purposes, but in the near future HR, Finance and Facilities will join. So we will have to add another portal server or two and an LB to manage the load.

We have one WF server and one secondary server that also runs the portal.
The ServiceManager and ServiceManagement DB's runs on the same SQL VM

As mentioned above, I saw a spike in CPU usage after upgrading to 9.1.2 but was unable to troubleshoot.

John_Long

Check out PR88950 in Cireson's open issues section. It seems to be what's bothering our system!

Jason_Meyer

Thanks John_Long, looking for that PR now. 

Working with Cireson Support on this, found we had some work item relationships that were no longer valid.  Cireson Support provided some SQL queries that helped identify these work items and we have cleaned them up.  We've turned off our temp. work around and so far (1 day) the 100% cpu utilization issue has not returned.

 

Carol_Lee

Hi, we are now on version 9.6.0. Had this issue immediately after the upgrade, because we had orphaned work items (discovered them when running the script in https://support.cireson.com/Problem/Edit/PR88950/). At that time, we were able to clean and solve the issue.

Now, we are having this CPU at 100% issue again, IIS Process Worker service is consuming most of it. Ran the same script in the database to look for orphaned work items, nothing returned. What could be happening? Please share your experience if you have any. Thanks

Gerhard_Goossens

Can you please share the script mentioned above? I do not have access to that PR. Will habt to put it in the archive for future reference.

We upgraded to 9.6 two weeks ago and have not had anny issues. Infact, comming from 9.2 I have seen a HUGE speed improvement in the portal.

EDIT: Ok, I found the script. Onhy had one and it will be pulled from the live DB on 15 December, so willl let sleeping dogs lay :-)

Carol_Lee

Hi @Gerhard_Goossens , I lost access to PR tickets in Cireson Support Portal too.

Here are the scripts I copied out on Nov 19.

;WITH Enum (EnumId, Field) AS

(SELECT DISTINCT StatusId [EnumId], 'StatusId'

  FROM WorkItem

UNION ALL

SELECT DISTINCT TRY_CAST(PriorityId as uniqueidentifier) [EnumId], 'PriorityId'

  FROM WorkItem

UNION ALL

SELECT DISTINCT CategoryId [EnumId], 'CategoryId'

  FROM WorkItem

UNION ALL

SELECT DISTINCT TierId [EnumId], 'TierId'

  FROM WorkItem

UNION ALL

SELECT DISTINCT SourceId [EnumId], 'SourceId'

  FROM WorkItem

UNION ALL

SELECT DISTINCT Resolution [EnumId], 'Resolution'

  FROM WorkItem

UNION ALL

SELECT DISTINCT Urgency [EnumId], 'Urgency'

  FROM WorkItem

UNION ALL

SELECT DISTINCT Impact [EnumId], 'Impact'

  FROM WorkItem

)

SELECT 'SELECT WorkItemId FROM WORKITEM WHERE ' + e.Field + ' = ''' + CAST(e.EnumId as nvarchar(36)) + ''''

FROM Enum e

LEFT OUTER JOIN DisplayString ds ON e.EnumId = ds.ElementID and ds.LocaleID='ENU'

WHERE ds.ElementID is NULL

and e.EnumId is not null

Carol_Lee

I have opened an Incident with Cireson. They were able to narrow down the issue to sorting "Active Work" by Priority. Still waiting for their further reply.