Older incidents not visible in Portal

Suleyman_Ozden

Hi

I am looking for someone to confirm that the reason older incidents are not showing in portal is due to the fact that they are moved to datawarehouse. We have set a 90 retention on incidents.

I can see cases in a search but when clicking on them I get an error.

Nicholas_Velich

To avoid confusion on one item @Brett_Moffett mentioned: "When the data is removed from the Service Manager DB the data should also be removed from the Cireson DB."

This does not occur at this time, and is why the items are appearing in your searches/grids: most searches/grids are pulling their content for the ServiceManagement (Cireson) database, and all forms are pulling their content from ServiceManager.

If you were to truncate the WorkItem table in your ServiceManagement table, the CacheBuilder would rebuild its contents with whatever items remain in ServiceManager, and these groomed items would disappear at that time. There are some more incremental ways of doing so via SQL query as well, as truncating can take a long time in larger organizations.

Brett_Moffett

I apologise @Suleyman_Ozden, I was under the impression that the cache builder did clean up the Cireson DB.
The fact that it doesn't explains why you are seeing the results that you see as the data remains in our DB and as a user opens the forms it will attempt to connect tot he ServiceManager DB to retrieve the data but it wont be there due to grooming.

Brian_Wiest

The retention time to me can be based on a few factors. 
Number of work items created within that time period, server/sql farm design and the number of workflows that are running. And Last SCSM 2012 R2 vs 2016

If you take a look at the blog regarding the processing improvements in 2016
https://blogs.technet.microsoft.com/servicemanager/2016/04/28/improved-entity-processing-in-service-manager-2016/
It also gives you some metrics on 2012 R2.

So I have my retention levels set at 90 days or less depending on the work item leaving me with ~27,000 work items in the active database. My first management server is just for workflow, all users connect to other management servers or web portal servers. Along with my SQL servers DB's running on SSD's

Peter_Settle

Suleyman_Ozden  We process around 1600 work items per week this includes SR's, IR's and CR's. This is all done on 2 load balanced cireson servers which are dedicated to cireson.

Brett_Moffett

When an Incident Work Item is created it is saved to the Service Manager database.
It is also synced to the Data Warehouse DB the next time the Extract, Transfer and Load (ETL) jobs run.

Incident work items remain in the Service Manager database until such time as they have their status set to CLOSED.
Once they have a status of closed, the retention settings from SCSM will count how many days it has been closed. After that time (in your case 90 days) the data will be deleted from the Service Manager database so the only record of it after those 90 days will be in the Data Warehouse DB.

The Cireson Portal uses a service (Called the Cireson Cache Builder) that syncs some key data from the Service Manager DB to the Cireson DB for use in the portal. When the data is removed from the Service Manager DB the data should also be removed from the Cireson DB.

As of version 7.3 of the portal we introduce Cireson Analytics that does a longer term storage of work items and other data but this is for reporting via dashboards in the portal and not wired to the search or Work Item usage in the portal at this stage.

I hope this covers what you are after and answers your question

Suleyman_Ozden

Thanks for confirming. The only thing I do not understand then is how come i can make a search for incidents that had a CLOSED date from september 2016? The search shows ID, title, affected user, status, assigned to user. I Wonder where it gets the information?

Our analyst are used to being able to lookup old cases. It does make sense to lookup how a particular case was resolved or check a users history of cases etc.

I Wonder if there is a way to provide that for them?

And also what the ideal ways to access the datawarehouse data would be? Power BI, Reporting services or something else?

I havent had the chance to look at the analytics yet.

Nicholas_Velich

To avoid confusion on one item @Brett_Moffett mentioned: "When the data is removed from the Service Manager DB the data should also be removed from the Cireson DB."

This does not occur at this time, and is why the items are appearing in your searches/grids: most searches/grids are pulling their content for the ServiceManagement (Cireson) database, and all forms are pulling their content from ServiceManager.

If you were to truncate the WorkItem table in your ServiceManagement table, the CacheBuilder would rebuild its contents with whatever items remain in ServiceManager, and these groomed items would disappear at that time. There are some more incremental ways of doing so via SQL query as well, as truncating can take a long time in larger organizations.

Brett_Moffett

I apologise @Suleyman_Ozden, I was under the impression that the cache builder did clean up the Cireson DB.
The fact that it doesn't explains why you are seeing the results that you see as the data remains in our DB and as a user opens the forms it will attempt to connect tot he ServiceManager DB to retrieve the data but it wont be there due to grooming.

Suleyman_Ozden

Thats ok.

I am planning to set the retention to 180 days. How is that going to effect performance? More data in Service ManagerDB, will mean more data that Cireson has to cache right?

Peter_Settle

We currently have all our work item set 365 days and suffer zero performance issue.

@Suleyman_Ozden

Brian_Wiest

The retention time to me can be based on a few factors. 
Number of work items created within that time period, server/sql farm design and the number of workflows that are running. And Last SCSM 2012 R2 vs 2016

If you take a look at the blog regarding the processing improvements in 2016
https://blogs.technet.microsoft.com/servicemanager/2016/04/28/improved-entity-processing-in-service-manager-2016/
It also gives you some metrics on 2012 R2.

So I have my retention levels set at 90 days or less depending on the work item leaving me with ~27,000 work items in the active database. My first management server is just for workflow, all users connect to other management servers or web portal servers. Along with my SQL servers DB's running on SSD's

Brett_Moffett

Totally agree with @Brian_Wiest on his comment.

Another thing to consider is our new product that was added to v7.3 of the portal that archives data from the Cireson DB and takes most of the WI data in to our Analytics database.
Depending on your reporting requirements, this feature might be enough data for you that would not need a data warehouse at all.

We did a series (USA, APAC and EMEA) of webinars on this in the past week and recordings of them are available at www.vimeo.com/teamcireson

Might be worth investigating.....

Suleyman_Ozden

Peter_Settle said:

We currently have all our work item set 365 days and suffer zero performance issue.

@Suleyman_Ozden

Hi Peter

Just to get an idea. How many Work Items do you have a month? And do users connect to workflow server?

I have a seperate server for Cireson. We have perhaps 1100-1200 Work items.

Peter_Settle

Suleyman_Ozden  We process around 1600 work items per week this includes SR's, IR's and CR's. This is all done on 2 load balanced cireson servers which are dedicated to cireson.

Brian_Wiest

@Brett_Moffett We are just got 7.3 running in our staging environment and these weekend will push to production. But one thing I wished it could do is that we will only get metrics from the live database moving forward. It will not pull the 2 years that are currently in our warehouse.
We also build our own web application that pulls the data into a custom SQL database to provide end user and analyst search options of all historical tickets, work logs, and configuration items.