Workflows not running consistently
I have a strange thing happening and I have run out of things to check
Problem:When users create IR or SR through the portal the workflow that changes the Support Group does not run every time.
Variables:Location: Location 1, Location 2 and Location 3
User Capacity: P for Staff and S for Students
Scenario:Our users are located in three locations and travel between locations. To accommodate the fact that they can be at any location at any time, they must select their current location when they complete the form in the portal.
This kicks off a workflow that changes the support group so that they can get support from where they are currently located.
Request Offering:In all the RO templates the Support Group is set to "StagingIT" along with the RO specific values such as Classification, priority etc.
Workflow Criteria:When IR or SR is created
Support Group = StagingIT
Capacity = P
Location = Location 1
This will then apply a template that sets the support Group to "Location 1 Staff Service Desk"
There are six similar workflows to accommodate the three locations and two user capacities.
Issue:As stated in the Problem statement, the above-mentioned workflows do not kick off consistently. About 1 in 10 created IR or SR does not work
Nothing has changed with the workflows, templates or ROs and this has all worked well since September last year
What did happen is that one of our VMWare hosts rebooted, the WF and SM DB server were guests on the host.
After the issue with the host was resolved and the guests started up automatically, I proceeded to shut down the portal server, orchestrator, DW Management server, WF server, SM DB and then the DW DB. I waited 5 minutes and started everything in reverse as listed above.
There are no other issues that I am aware of and all other workflows are running ie WIs status changes from new to in progress or active, notifications get sent out etc
Troubleshooting:I went through the OpsMan and other event logs and there are no errors or warnings.
Ran the SQL script to check the workflow status and how many minutes behind they are.
Removed and recreated the six workflows.
Conclusion:What I suspect is that the order of workflows when something gets created has been messed up and that some update workflow is running before the ones I need and then the WI is not in "Created" state anymore.
Currently, we are manually moving the WIs to the correct Support Group while I search for a solution.
I have started to build an Orchestrator workflow to check for newly created WIs and then do the support Group changes, but imo that is only a workaround for perhaps a larger issue.
At this stage, I have read anything I can possibly find to find a solution
If anyone is willing to throw some ideas to troubleshoot it will be highly appreciated.