Home General Discussion

Anyone else seeing 100% CPU usage on web servers with Cireson Portal version 9.1.2?

Jason_MeyerJason_Meyer Customer Advanced IT Monkey ✭✭✭
edited May 2019 in General Discussion
Anyone else seeing 100% CPU usage with Portal version 9.1.2?   We upgraded last night and roughly 18 hours later both our web front end servers are pegged at 100% CPU.   We are preparing to roll back to the previous version.

Both servers have been rebooted and have returned to 100% CPU.   The website is working but very slow.

Any triage or troubleshooting that we can do?

Answers

  • Brian_WiestBrian_Wiest Customer Super IT Monkey ✭✭✭✭✭
    What service is eating up the CPU?
  • Jason_MeyerJason_Meyer Customer Advanced IT Monkey ✭✭✭
    IIS.
  • Shane_WhiteShane_White Cireson Support Super IT Monkey ✭✭✭✭✭
    @Jason_Meyer

    How many analysts/end users do you have using the Portal? It could be something to do with the sessions not ending.

    In session state in IIS what time have you got it set to?

    Thanks,
    Shane.
  • Gerhard_GoossensGerhard_Goossens Customer Advanced IT Monkey ✭✭✭
    I also had this issue on my test VM, downgraded back to 9.1.0
  • Brian_WiestBrian_Wiest Customer Super IT Monkey ✭✭✭✭✭
    After downgrading did you perform an iisreset /restart?
    I am running v9.2.1.2016 on Server 2016 and not seeing any performance issues.
  • Gerhard_GoossensGerhard_Goossens Customer Advanced IT Monkey ✭✭✭
    @Brian_Wiest
    Nope, just reinstalled the old version and everything was back to normal.
    No IIS/CacheBuilder or VM bounce

    Still wanted to investigate but had to rollback for other testing.
  • Jason_MeyerJason_Meyer Customer Advanced IT Monkey ✭✭✭

    @Shane_White - We have about 15k users, 1k analysts.  It was running 9.1.1 with no issues for 4-5 months..  Day after the upgrade to 9.1.2, after about 2 hours of load it went to 100% CPU.  Asking our engineers about the session state time question.

    We rolled back as well and are monitoring the system today.

  • Shane_WhiteShane_White Cireson Support Super IT Monkey ✭✭✭✭✭
    @Jason_Meyer

    I would advise putting in a Support Ticket for this but in the meantime it may be a performance issue with big environments.

    @Brian_Wiest / @Gerhard_Goossens
    how many users and analysts do you have?

    Thanks,
    Shane.
  • Brian_WiestBrian_Wiest Customer Super IT Monkey ✭✭✭✭✭
    In production about 1K analysts and ~25K users. But for production we are running 5 Portal servers. 1 is just for the cache builder and the other 4 are connected to the NLB.
    Currently in the UAT we have only about 10-100 testing depending on what is getting UAT approval to move to production.
    Currently we are at 0.1% on IIS Worker process.
  • Shane_WhiteShane_White Cireson Support Super IT Monkey ✭✭✭✭✭
    @Jason_Meyer

    How many servers are you running? When you upgraded, did you check for any errors in the logs? Or in event viewer?

    Thanks,
    Shane.
  • Gerhard_GoossensGerhard_Goossens Customer Advanced IT Monkey ✭✭✭
    @Jason_Meyer

    I would advise putting in a Support Ticket for this but in the meantime it may be a performance issue with big environments.

    @Brian_Wiest / @Gerhard_Goossens
    how many users and analysts do you have?

    Thanks,
    Shane.
    We have 60k users and about 150 analysts. I would guess that about 1000 of the users use the system on a monthly basis.

    For now, the system is only used for IT purposes, but in the near future HR, Finance and Facilities will join. So we will have to add another portal server or two and an LB to manage the load.

    We have one WF server and one secondary server that also runs the portal.
    The ServiceManager and ServiceManagement DB's runs on the same SQL VM

    As mentioned above, I saw a spike in CPU usage after upgrading to 9.1.2 but was unable to troubleshoot.




  • John_LongJohn_Long Customer Advanced IT Monkey ✭✭✭
    edited October 2019

    Check out PR88950 in Cireson's open issues section. It seems to be what's bothering our system!

  • Jason_MeyerJason_Meyer Customer Advanced IT Monkey ✭✭✭
    edited October 2019

    Thanks John_Long, looking for that PR now. 

    Working with Cireson Support on this, found we had some work item relationships that were no longer valid.  Cireson Support provided some SQL queries that helped identify these work items and we have cleaned them up.  We've turned off our temp. work around and so far (1 day) the 100% cpu utilization issue has not returned.

     

  • Carol_LeeCarol_Lee Customer IT Monkey ✭
    edited November 2019

    Hi, we are now on version 9.6.0. Had this issue immediately after the upgrade, because we had orphaned work items (discovered them when running the script in https://support.cireson.com/Problem/Edit/PR88950/). At that time, we were able to clean and solve the issue.

    Now, we are having this CPU at 100% issue again, IIS Process Worker service is consuming most of it. Ran the same script in the database to look for orphaned work items, nothing returned. What could be happening? Please share your experience if you have any. Thanks

  • Gerhard_GoossensGerhard_Goossens Customer Advanced IT Monkey ✭✭✭
    edited November 2019

    Can you please share the script mentioned above? I do not have access to that PR. Will habt to put it in the archive for future reference.

    We upgraded to 9.6 two weeks ago and have not had anny issues. Infact, comming from 9.2 I have seen a HUGE speed improvement in the portal.


    EDIT: Ok, I found the script. Onhy had one and it will be pulled from the live DB on 15 December, so willl let sleeping dogs lay :-)

  • Carol_LeeCarol_Lee Customer IT Monkey ✭

    Hi @Gerhard_Goossens , I lost access to PR tickets in Cireson Support Portal too.

    Here are the scripts I copied out on Nov 19.

    ;WITH Enum (EnumId, Field) AS

    (SELECT DISTINCT StatusId [EnumId], 'StatusId'

      FROM WorkItem

    UNION ALL

    SELECT DISTINCT TRY_CAST(PriorityId as uniqueidentifier) [EnumId], 'PriorityId'

      FROM WorkItem

    UNION ALL

    SELECT DISTINCT CategoryId [EnumId], 'CategoryId'

      FROM WorkItem

    UNION ALL

    SELECT DISTINCT TierId [EnumId], 'TierId'

      FROM WorkItem

    UNION ALL

    SELECT DISTINCT SourceId [EnumId], 'SourceId'

      FROM WorkItem

    UNION ALL

    SELECT DISTINCT Resolution [EnumId], 'Resolution'

      FROM WorkItem

    UNION ALL

    SELECT DISTINCT Urgency [EnumId], 'Urgency'

      FROM WorkItem

    UNION ALL

    SELECT DISTINCT Impact [EnumId], 'Impact'

      FROM WorkItem

    )

    SELECT 'SELECT WorkItemId FROM WORKITEM WHERE ' + e.Field + ' = ''' + CAST(e.EnumId as nvarchar(36)) + ''''

    FROM Enum e

    LEFT OUTER JOIN DisplayString ds ON e.EnumId = ds.ElementID and ds.LocaleID='ENU'

    WHERE ds.ElementID is NULL

    and e.EnumId is not null

  • Carol_LeeCarol_Lee Customer IT Monkey ✭

    I have opened an Incident with Cireson. They were able to narrow down the issue to sorting "Active Work" by Priority. Still waiting for their further reply.

Sign In or Register to comment.