Workflows not running consistently

Gerhard_Goossens

Good day everyone,

I have a strange thing happening and I have run out of things to check

Problem:

When users create IR or SR through the portal the workflow that changes the Support Group does not run every time.

Variables:

Location: Location 1, Location 2 and Location 3
User Capacity: P for Staff and S for Students

Scenario:

Our users are located in three locations and travel between locations. To accommodate the fact that they can be at any location at any time, they must select their current location when they complete the form in the portal. 
This kicks off a workflow that changes the support group so that they can get support from where they are currently located.

Request Offering:

In all the RO templates the Support Group is set to "StagingIT" along with the RO specific values such as Classification, priority etc.

Workflow Criteria:

When IR or SR is created
Support Group = StagingIT
Capacity = P
Location = Location 1

This will then apply a template that sets the support Group to "Location 1 Staff Service Desk"

There are six similar workflows to accommodate the three locations and two user capacities.

Issue:

As stated in the Problem statement, the above-mentioned workflows do not kick off consistently. About 1 in 10 created IR or SR does not work

Nothing has changed with the workflows, templates or ROs and this has all worked well since September last year

What did happen is that one of our VMWare hosts rebooted, the WF and SM DB server were guests on the host.

After the issue with the host was resolved and the guests started up automatically, I proceeded to shut down the portal server, orchestrator, DW Management server, WF server, SM DB and then the DW DB. I waited 5 minutes and started everything in reverse as listed above.

There are no other issues that I am aware of and all other workflows are running ie WIs status changes from new to in progress or active, notifications get sent out etc

Troubleshooting:

I went through the OpsMan and other event logs and there are no errors or warnings.
Ran the SQL script to check the workflow status and how many minutes behind they are.
Removed and recreated the six workflows.

Conclusion:

What I suspect is that the order of workflows when something gets created has been messed up and that some update workflow is running before the ones I need and then the WI is not in "Created" state anymore.

Currently, we are manually moving the WIs to the correct Support Group while I search for a solution.

I have started to build an Orchestrator workflow to check for newly created WIs and then do the support Group changes, but imo that is only a workaround for perhaps a larger issue.

At this stage, I have read anything I can possibly find to find a solution
If anyone is willing to throw some ideas to troubleshoot it will be highly appreciated.

Regards
Gerhard

Chris_Chekaluk1

Not sure if it would help or not, but I've got a SQL query against the ServiceManager db I have used occasionally whose purpose is to show which native SCSM workflows hit a given IR or SR, and when they do.  Let me know if you think this may help.

Brian_Wiest

Recommend to clear the SCSM cache.

https://community.cireson.com/discussion/3905/full-scsm-and-cireson-cache-flush-powershell-script

Gerhard_Goossens

Brian_Wiest said:

Recommend to clear the SCSM cache.

https://community.cireson.com/discussion/3905/full-scsm-and-cireson-cache-flush-powershell-script

There are no issues with syncing between SM and the portal, it gets synced within 1m. I have also stopped the SM services and removed the HealthState in the SM installation folder.

But the provided script seems like an awesome tool to refresh the portal side. Will definitely give it a shot (once I get this sorted out...)

Jeff_Lang

@Gerhard_Goossens

I have seen this before, and sometimes have had success in resolving it by deleting the Service Manager Health State Folder, this gets recreated when Service manager is started again. In saying that i have seen it at times when removing this folder did not help as well.

Chris_Chekaluk1

Not sure if it would help or not, but I've got a SQL query against the ServiceManager db I have used occasionally whose purpose is to show which native SCSM workflows hit a given IR or SR, and when they do.  Let me know if you think this may help.

Gerhard_Goossens

Chris_Chekaluk1 said:

Not sure if it would help or not, but I've got a SQL query against the ServiceManager db I have used occasionally whose purpose is to show which native SCSM workflows hit a given IR or SR, and when they do.  Let me know if you think this may help.

@Chris_Chekaluk1

That sounds awesome, please send me the query. I'm sure it will be useful.

Regards
Gerhard

Chris_Chekaluk1

@Gerhard_Goossens: sorry for delay, when I looked at the query, I realized it was more for notification subscriptions, and wasn't returning non-notification workflows.  Looking into updating this as I can.

Chris_Chekaluk1

@Gerhard_Goossens:  I suspect my issue was probably due more to job status records getting groomed out of database for older work items.  The below query works for a 'newer' SR and should work for all WI classes since the I am looking in the BaseManagedEntity table for the work item id...

use ServiceManager

declare @id as varchar(25)

set @id = 

'SR1107473'

declare @lang as varchar(3)

set @lang = 'enu'

SELECT top 1000

bme.name as 'Id'

-- ,bme.BaseManagedEntityId

,r.DisplayName as 'Rule'

,isnull(wfstatus.DataItem,'') as  'DataItem'

,isnull(wfstatus.ErrorMessage,'') as 'Workflow Error Message'

,wfstatus.LastModified

,wfstatus.DataItemLastModified

,wfstatus.Processed

-- ,wfstatus.RowId

-- ,jStatus.JobId

,jStatus.Status

,cast(jStatus.Output as xml) as 'Job Status Output'

,jStatus.ErrorCode

,isnull(jStatus.ErrorMessage,'') as 'Job Status Error Message'

FROM WindowsWorkflowTaskJobStatus WFStatus

  join RuleView R on (R.Id = WFStatus.RuleId and r.LanguageCode = @lang)

  join BaseManagedEntity bme on (bme.BaseManagedEntityId = wfstatus.BaseManagedEntityId)

  join JobStatus jStatus on (jStatus.BatchId = WFStatus.BatchId)

  

where bme.Name = @id

  

order by wfstatus.LastModified  asc