Anyone else seeing 100% CPU usage on web servers with Cireson Portal version 9.1.2?
Both servers have been rebooted and have returned to 100% CPU. The website is working but very slow.
Any triage or troubleshooting that we can do?
AGREEMENT BETWEEN USER AND CIRESON
The Cireson Community Web Site is comprised of various web pages operated by Cireson, LLC (collectively, the web pages are referred to herein as the “Cireson Community Web Site”).
Cireson reserves the right to change the terms, conditions, and notices under which the Cireson Community Web Site is offered, including but not limited to the charges associated with the use of the Cireson Community Web Site upon written notice.
LINKS TO THIRD PARTY SITES
The Cireson Community Web Site may contain links to other Web Sites (“Linked Sites”). The Linked Sites are not under the control of Cireson and therefore Cireson is not responsible for the contents of any Linked Site, including, without limitation, any link contained in a Linked Site, or any changes or updates to a Linked Site. Cireson is not responsible for webcasting or any other form of transmission received from any Linked Site. Cireson is providing these links to you only as a convenience, and the inclusion of any link does not imply endorsement by Cireson of the site or any association with its operators.
NO UNLAWFUL OR PROHIBITED USE
As a condition of your use of the Cireson Community Web Site, you warrant to Cireson that you will not use the Cireson Community Web Site for any purpose that is prohibited by law, rule regulation or by any of these terms, conditions, and notices, including, without limitation the export of any Software or technical data to any country prohibited by law. You may not use the Cireson Community Web Site in any manner which could damage, disable, overburden, or impair the Cireson Community Web Site or interfere with any other party’s use and enjoyment of the Cireson Community Web Site. You may not obtain or attempt to obtain any materials or information through any means not intentionally made available or provided for through the Cireson Community Web Site.
USE OF COMMUNICATION SERVICES
The Cireson Community Web Site may contain bulletin board services, chat areas, news groups, forums, communities, personal web pages, calendars, and/or other message or communication facilities designed to enable you to communicate with the public at large or with a group (each a “Communication Service,” and collectively, “Communication Services”). You agree to use the Communication Services only to post, send and receive messages and material that are appropriate, proper and related to the particular Communication Service. By way of example, and not as a limitation, you agree that when using a Communication Service, you will not:
Cireson has no obligation to monitor the Communication Services. However, Cireson reserves the right to review materials posted to a Communication Service and to remove any materials in its sole discretion. Cireson reserves the right to suspend or terminate your access to any or all of the Communication Services at any time without notice for any reason whatsoever.
Cireson reserves the right at all times to disclose any information as necessary to satisfy any applicable law, rule, regulation, legal process or governmental request, or to edit, refuse to post or to remove any information or materials, in whole or in part, in Cireson’s sole discretion.
Always use caution when giving out any personally identifying information about yourself or your children in any Communication Service. Always use caution when giving out any information concerning your employer or customers, if any. Cireson does not control or endorse the content, messages or information found in any Communication Service and, therefore, Cireson specifically disclaims any liability with regard to the Communication Services and any actions resulting from your participation in any Communication Service. Managers and hosts are not authorized Cireson spokespersons, and their views do not necessarily reflect those of Cireson.
Materials uploaded to a Communication Service may be subject to posted limitations on usage, reproduction and/or dissemination. You are responsible for adhering to such limitations if you download the materials.
MATERIALS PROVIDED TO CIRESON OR POSTED ON THE CIRESON COMMUNITY WEB SITE
Cireson does not claim ownership of the materials you provide to Cireson via the Cireson Community Web Site (including feedback and suggestions) or post, upload, input or submit to the Cireson Community Web Site or its associated services (collectively “Submissions”). However, by posting, uploading, inputting, providing or submitting your Submission you are granting Cireson, its affiliated companies and sublicensees permission to use your Submission in connection with the operation of their Internet businesses including, without limitation, the rights to: copy, distribute, transmit, publicly display, publicly perform, reproduce, edit, translate and reformat your Submission; and to publish your name in connection with your Submission.
No compensation will be paid with respect to the use of your Submission, as provided herein. Cireson is under no obligation to post or use any Submission you may provide and may remove any Submission at any time in Cireson’s sole discretion.
By posting, uploading, inputting, providing or submitting your Submission you warrant and represent that you own or otherwise control all of the rights to your Submission as described in this section including, without limitation, all the rights necessary for you to provide, post, upload, input or submit the Submissions.
THE INFORMATION, SOFTWARE, PRODUCTS, AND SERVICES INCLUDED IN OR AVAILABLE THROUGH THE CIRESON COMMUNITY WEB SITE MAY INCLUDE INACCURACIES OR TYPOGRAPHICAL ERRORS. CHANGES ARE PERIODICALLY ADDED TO THE INFORMATION HEREIN. CIRESON (OR ITS SUPPLIERS) MAY MAKE IMPROVEMENTS AND/OR CHANGES IN THE CIRESON COMMUNITY WEB SITE AT ANY TIME. ADVICE RECEIVED VIA THE CIRESON COMMUNITY WEB SITE SHOULD NOT BE RELIED UPON FOR PERSONAL, MEDICAL, LEGAL OR FINANCIAL DECISIONS AND YOU SHOULD CONSULT AN APPROPRIATE PROFESSIONAL FOR SPECIFIC ADVICE TAILORED TO YOUR SITUATION.
NEITHER CIRESON NOR ANY OF ITS SUPPLIERS MAKE ANY REPRESENTATIONS ABOUT THE SUITABILITY, RELIABILITY, AVAILABILITY, TIMELINESS, AND ACCURACY OF THE INFORMATION, SOFTWARE, PRODUCTS, SERVICES AND RELATED GRAPHICS CONTAINED ON THE CIRESON COMMUNITY WEB SITE FOR ANY PURPOSE. TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, ALL SUCH INFORMATION, SOFTWARE, PRODUCTS, SERVICES AND RELATED GRAPHICS ARE PROVIDED “AS IS” WITHOUT WARRANTY OR CONDITION OF ANY KIND. CIRESON AND/OR ITS SUPPLIERS HEREBY DISCLAIMS ALL WARRANTIES AND CONDITIONS WITH REGARD TO THIS INFORMATION, SOFTWARE, PRODUCTS, SERVICES AND RELATED GRAPHICS, INCLUDING ALL IMPLIED WARRANTIES OR CONDITIONS OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT.
SERVICE CONTACT: [email protected]
Cireson reserves the right, in its sole discretion, to terminate your access to the Cireson Community Web Site and the related services or any portion thereof at any time, without notice.
To the maximum extent permitted by law, this agreement is governed by the laws of the State of California, U.S.A. and you hereby consent to the exclusive jurisdiction and venue of courts in San Diego County, California, U.S.A. in all disputes arising out of or relating to the use of the Cireson Community Web Site. Use of the Cireson Community Web Site is unauthorized in any jurisdiction that does not give effect to all provisions of these terms and conditions, including without limitation this paragraph. You agree that no joint venture, partnership, employment, or agency relationship exists between you and Cireson as a result of this agreement or use of the Cireson Community Web Site. Cireson’s performance of this agreement is subject to existing laws and legal process, and nothing contained in this agreement is in derogation of Cireson’s right to comply with governmental, court and law enforcement requests or requirements relating to your use of the Cireson Community Web Site or information provided to or gathered by Cireson with respect to such use. If any part of this agreement is determined to be invalid or unenforceable pursuant to applicable law including, but not limited to, the warranty disclaimers and liability limitations set forth above, then the invalid or unenforceable provision will be deemed superseded by a valid, enforceable provision that most closely matches the intent of the original provision and the remainder of the agreement shall continue in effect. Unless otherwise specified herein, this agreement constitutes the entire agreement between the user and Cireson with respect to the Cireson Community Web Site and it supersedes all prior or contemporaneous communications and proposals, whether electronic, oral or written, between the user and Cireson with respect to the Cireson Community Web Site. A printed version of this agreement and of any notice given in electronic form shall be admissible in judicial or administrative proceedings based upon or relating to this agreement to the same extent and subject to the same conditions as other business documents and records originally generated and maintained in printed form. It is the express wish to the parties that this agreement and all related documents be drawn up in English.
COPYRIGHT AND TRADEMARK NOTICES:
All contents of the Cireson Community Web Site are: Copyright 2016 by Cireson, LLC and/or its suppliers. All rights reserved.
The names of actual companies and products mentioned herein and on the Cireson Community Web Site may be the trademarks of their respective owners.
The example companies, organizations, products, people and events depicted herein are fictitious. No association with any real company, organization, product, person, or event is intended or should be inferred.
Any rights not expressly granted herein are reserved.
NOTICES AND PROCEDURE FOR MAKING CLAIMS OF COPYRIGHT INFRINGEMENT
Cireson respects the intellectual property of others and asks that users of the Cireson Community Web Site do the same. In connection with the Cireson Community Web Site, we have adopted and implemented a policy respecting copyright law that provides for the removal of any infringing materials and for the termination, in appropriate circumstances, of users of the Cireson Community Web Site who are repeat infringers of intellectual property rights, including copyrights. If you believe that one of the Cireson Community Web Site users is, through the use of the Cireson Community Web Site, unlawfully infringing the copyright(s) in a work, and wish to have the allegedly infringing material removed, the following information in the form of a written notification (pursuant to 17 U.S.C. § 512(c)) must be provided to our designated Copyright Agent:
Please note that, pursuant to 17 U.S.C. § 512(f), any misrepresentation of material fact (falsities) in a written notification automatically subjects the complaining party to liability for any damages, costs and attorney’s fees incurred by us in connection with the written notification and allegation of copyright infringement.
Designated Agent: Matthew W. Deen, Esq.
Address of Agent: 1620 Fifth Avenue, Suite 875, San Diego, CA 92101
Email: [email protected]
How many analysts/end users do you have using the Portal? It could be something to do with the sessions not ending.
In session state in IIS what time have you got it set to?
Nope, just reinstalled the old version and everything was back to normal.
No IIS/CacheBuilder or VM bounce
Still wanted to investigate but had to rollback for other testing.
@Shane_White - We have about 15k users, 1k analysts. It was running 9.1.1 with no issues for 4-5 months.. Day after the upgrade to 9.1.2, after about 2 hours of load it went to 100% CPU. Asking our engineers about the session state time question.
We rolled back as well and are monitoring the system today.
I would advise putting in a Support Ticket for this but in the meantime it may be a performance issue with big environments.
@Brian_Wiest / @Gerhard_Goossens
how many users and analysts do you have?
How many servers are you running? When you upgraded, did you check for any errors in the logs? Or in event viewer?
For now, the system is only used for IT purposes, but in the near future HR, Finance and Facilities will join. So we will have to add another portal server or two and an LB to manage the load.
We have one WF server and one secondary server that also runs the portal.
The ServiceManager and ServiceManagement DB's runs on the same SQL VM
As mentioned above, I saw a spike in CPU usage after upgrading to 9.1.2 but was unable to troubleshoot.
Check out PR88950 in Cireson's open issues section. It seems to be what's bothering our system!
Thanks John_Long, looking for that PR now.
Working with Cireson Support on this, found we had some work item relationships that were no longer valid. Cireson Support provided some SQL queries that helped identify these work items and we have cleaned them up. We've turned off our temp. work around and so far (1 day) the 100% cpu utilization issue has not returned.
Hi, we are now on version 9.6.0. Had this issue immediately after the upgrade, because we had orphaned work items (discovered them when running the script in https://support.cireson.com/Problem/Edit/PR88950/). At that time, we were able to clean and solve the issue.
Now, we are having this CPU at 100% issue again, IIS Process Worker service is consuming most of it. Ran the same script in the database to look for orphaned work items, nothing returned. What could be happening? Please share your experience if you have any. Thanks
Can you please share the script mentioned above? I do not have access to that PR. Will habt to put it in the archive for future reference.
We upgraded to 9.6 two weeks ago and have not had anny issues. Infact, comming from 9.2 I have seen a HUGE speed improvement in the portal.
EDIT: Ok, I found the script. Onhy had one and it will be pulled from the live DB on 15 December, so willl let sleeping dogs lay :-)
Hi @Gerhard_Goossens , I lost access to PR tickets in Cireson Support Portal too.
Here are the scripts I copied out on Nov 19.
;WITH Enum (EnumId, Field) AS
(SELECT DISTINCT StatusId [EnumId], 'StatusId'
SELECT DISTINCT TRY_CAST(PriorityId as uniqueidentifier) [EnumId], 'PriorityId'
SELECT DISTINCT CategoryId [EnumId], 'CategoryId'
SELECT DISTINCT TierId [EnumId], 'TierId'
SELECT DISTINCT SourceId [EnumId], 'SourceId'
SELECT DISTINCT Resolution [EnumId], 'Resolution'
SELECT DISTINCT Urgency [EnumId], 'Urgency'
SELECT DISTINCT Impact [EnumId], 'Impact'
SELECT 'SELECT WorkItemId FROM WORKITEM WHERE ' + e.Field + ' = ''' + CAST(e.EnumId as nvarchar(36)) + ''''
FROM Enum e
LEFT OUTER JOIN DisplayString ds ON e.EnumId = ds.ElementID and ds.LocaleID='ENU'
WHERE ds.ElementID is NULL
and e.EnumId is not null
I have opened an Incident with Cireson. They were able to narrow down the issue to sorting "Active Work" by Priority. Still waiting for their further reply.