Anyone else having issues with PowerShell Activities getting "stuck" in progress?

Raul_Noa

Occasionally, PSAs will hang indefinitely in an "In Progress" state. As a workaround, we find the ones that have been in progress for over an hour and set the status to "Rerun" and they always successfully execute on a subsequent run, usually in less than a minute. I've found these two variations of errors in the PSA logs for the ones that get stuck in progress, and corresponding errors in the workflow server's windows application event log.


Unexpected Error 1:  (most predominant for stuck PSAs)
======================================

3/28/2018 7:55:12 AM Invoke PowerShell Object

3/28/2018 7:56:56 AM PowerShell finish executing

3/28/2018 7:56:56 AM Printing PowerShell output to Output property

3/28/2018 7:56:58 AM Error excuting PowerShell

3/28/2018 7:56:58 AM The specified value did not conform to the maximum length requirements.
======================================
Corresponding Windows Event Logs on Workflow Server:

======================================

A Windows Workflow Foundation workflow failed during execution.   Workflow Type: Cireson.Powershell.Activity.Workflow.RunPowershellScript  Workflow Identifier: 81fd4086-4496-d1b8-1657-5dff1f7e25f0  Exception Type: Microsoft.EnterpriseManagement.Common.InvalidSimpleObjectValueException  Exception Message: The specified value did not conform to the maximum length requirements.   Exception Stack:    at Cireson.Powershell.Activity.Workflow.RunPowershellScript.a(Object A_0, EventArgs A_1)    at System.Workflow.ComponentModel.Activity.RaiseEvent(DependencyProperty dependencyEvent, Object sender, EventArgs e)    at System.Workflow.Activities.CodeActivity.Execute(ActivityExecutionContext executionContext)    at System.Workflow.ComponentModel.ActivityExecutor`1.Execute(T activity, ActivityExecutionContext executionContext)    at System.Workflow.ComponentModel.ActivityExecutorOperation.Run(IWorkflowCoreRuntime workflowCoreRuntime)    at System.Workflow.Runtime.Scheduler.Run()

======================================

Unexpected Error 2: (less common, but also seen with stuck PSAs)
======================================

3/28/2018 10:09:38 AM Invoke PowerShell Object

3/28/2018 10:11:18 AM PowerShell finish executing

3/28/2018 10:11:18 AM Printing PowerShell output to Output property

3/28/2018 10:11:18 AM Error excuting PowerShell

3/28/2018 10:11:18 AM Exception of type 'System.OutOfMemoryException' was thrown.

======================================
Corresponding Windows Event Logs on Workflow Server:

======================================

Application: MonitoringHost.exe

Framework Version: v4.0.30319

Description: The process was terminated due to an unhandled exception.

Exception Info: System.OutOfMemoryException

Stack:

   at System.Workflow.Runtime.Hosting.DefaultWorkflowSchedulerService+WorkItem.Invoke(System.Workflow.Runtime.Hosting.WorkflowSchedulerService)

   at System.Workflow.Runtime.Hosting.DefaultWorkflowSchedulerService.QueueWorkerProcess(System.Object)

   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()

   at System.Threading.ThreadPoolWorkQueue.Dispatch()

======================================

Faulting application name: MonitoringHost.exe, version: 7.1.10184.0, time stamp: 0x522a23d5

Faulting module name: KERNELBASE.dll, version: 6.3.9600.18895, time stamp: 0x5a4b1cf7

Exception code: 0xe0434352

Fault offset: 0x00000000000092fc

Faulting process id: 0x56f4

Faulting application start time: 0x01d3c60364f18230

Faulting application path: \Program Files\Microsoft System Center 2012 R2\Service Manager\MonitoringHost.exe

Faulting module path: C:\Windows\system32\KERNELBASE.dll

Report Id: 53c029e6-31f7-11e8-811b-005056aa6011

Faulting package full name: 

Faulting package-relative application ID: 

======================================


By contrast:

Normal and Expected Error: If the script actually executes and produces an error we see a different type of error, as expected and the PSA changes to a Failed status.
======================================

3/28/2018 10:06:38 AM Invoke PowerShell Object

3/28/2018 10:06:38 AM PowerShell finish executing

3/28/2018 10:06:38 AM Printing PowerShell output to Output property

3/28/2018 10:06:38 AM Assigning ManuallyRunWorkflow to false

3/28/2018 10:06:38 AM Powershell has error.

Cannot bind argument to parameter 'Id' because it is null.Cannot bind argument to parameter 'BySource' because it is null.

3/28/2018 10:06:38 AM Start updating instance

3/28/2018 10:06:38 AM Instance successfully updated
======================================




I do have a support ticket open with Cireson, just wanted to see if anyone else is experiencing this.

Thanks!

Daniel_Polivka1

Oh boy, responding to a 6 year old thread. I also have this issue, randomly a PSA/Cloud Activity with a PSA will just hang In Progress. Here's an error dump from me if it helps. Anyone else experience this? I'd love to either know of a fix or a workaround.

Microsoft.EnterpriseManagement.Common.InvalidSimpleObjectValueException: The specified value did not conform to the maximum length requirements.
at Microsoft.EnterpriseManagement.Configuration.ManagementPackProperty.ValidateValue(Object value, EnterpriseManagementGroup managementGroup)
at Microsoft.EnterpriseManagement.Common.EnterpriseManagementSimpleObject.set_Value(Object value)
at Cireson.Cloud.Activity.Workflow.Classes.PowershellService.InvokePSA(EnterpriseManagementObjectProjection emop, Guid _instanceTrigerId, Boolean isConfiguredWorkflow)
at Cireson.Cloud.Activity.Workflow.ProcessCiresonCloudActivity.OCAvC6tRY(EnterpriseManagementGroup , Guid , Guid , Boolean )
at Cireson.Cloud.Activity.Workflow.ProcessCiresonCloudActivity.Execute(ActivityExecutionContext CiresonCloudActivityContext)
at System.Workflow.ComponentModel.ActivityExecutorOperation.Run(IWorkflowCoreRuntime workflowCoreRuntime)
at System.Workflow.Runtime.Scheduler.Run()

Geoff_Ross

Hi Daniel,

I think your error is the Output property on the PSA / Cloud Activity getting too long. The next line to be written is pushing it over the max length of the field causing the

 The specified value did not conform to the maximum length requirements.

Are you deliberately writing to the output a lot?

Geoff

Simon_Zeinhofer

@Daniel_Polivka1 I guess for the stuck Activities, there is a PR open. As far as I know, and based on the feedback, this happens when 2 or more PSAs are created the same time (or shall start at the same time, I don't remember correctly). We have the same issue and were thinking about a custom workflow to correct that - But I guess it is more save to wait for a official bugfix :)

We also have this issue with the max length error from time to time. And as written below I cannot imagine, that the output has so many characters in that particular script. But in our case, the script runs through, but as the output is too long, the workflow cannot set it to Completed. Do you experience the same?

Simon_Zeinhofer

@Geoff_Ross We have the issue with the

The specified value did not conform to the maximum length requirements. 

error in a certain Powershell Activity as well recently. The thing is, we only write very little information to the output here, so I cannot imagine being over the 65536 characters length. Might there be another issue?

Daniel_Polivka1

@Geoff_Ross Yes there is definitely the possibility of my outputs becoming quite large on this particular PSA. I normally utilize Start/Stop-Transcript on my scripts, but I notice the output for the PSAs only write to the Output tab, not my "saved off" location too as specified in the script.

Daniel_Polivka1

@Simon_Zeinhofer Revisiting your earlier comment, I've seen this happen on templates that have more than one automation activity in the workflow, and it honestly feels like the hang up is completely random. For example, sometimes it hangs on an activity that is a PowerShell script that is close to 1000 lines long and does some complex tasks, and sometimes it doesn't. And, sometimes it hangs on a script that is only 4 lines long and performs one very simple task, and sometimes it doesn't. The randomness of this issue makes troubleshooting complicated. I am utilizing Cloud Activities that run a PowerShell script, so I'm wondering if I should just re-do all of my Cloud Activities as PowerShell activities instead.

Daniel_Polivka1

I should probably also mention I've experienced this in a multitude of different ways. For example, I originially thought the activity was executing, running the script, and then was unable to switch status to Completed. So, I crafted a scheduled task that would identify these activities that were In Progress for an abnormally long time, and set the activity to Skipped, allowing the workflow to continue. However, doing this led me to a new discovery, which was that the script inside the activity wasn't actually executing 100% of the time. And this discovery led me down a deeper rabbit hole, and I found that in most (not all!) of the cases where the script would not execute, sometimes there wouldn't even be the script present in the activity at all! So like for this screenshot, I found an example of the same script being utilized in two different activities, and one has the script ID and the script, and the other has the script ID, but no script!

And then of course this rabbit hole kept going and there are times where neither the script ID or script are present. All of this is seemingly random, with nothing else correlating at those failure times, like I've wondered if workflows are failing, or I have connection issues, etc… but so far I have no overt correlation, even checking Event Viewer logs around those times.

Simon_Zeinhofer

@Daniel_Polivka1

We have the same issue from time to time. The PSA is running, but never completes and the status stays "In Progress", the output property is empty as well. The thing is, we know that the PSA has been running, as the activity sets some values in MEMCM and SCSM to install our clients, and without that script, we couldn't start it.

I haven't been looking inside the OpsManager Logs but for me it looks like a workflow error. As soon as it happens again, I will have a look at this log and give you a feedback.

Geoff_Ross

@Simon_Zeinhofer
Myself and Adam caught up with Dan last week and we think we found the issue.
If the output is too long for the Output property (4000 chars) it fails to set the output but then crashes so doesn't do the final step of setting the activity to complete.

We are working on a fix.

Geoff

Simon_Zeinhofer

Cool to hear that :)

Stupid question from my side: When i look inside the class definition, the output property has a max size of 65536 - so are the 4000 max length restrictions from SCSM itself?

Simon_Zeinhofer

@Geoff_Ross is there any update on that issue? Right now we start many install requests for new clients and these consist mostly of PSAs. The output is empty and the activity stays In Progress, but we know that the script itself has been executed. As we really use PSAs and not Cloud Activities, I don't think any of these activities has an output with more than the max length. Has anyone else experienced the same issue?