Change Request Hangs/issues with attachments

Sean_Terry

We are running 2016 SCSM UR4 and we have issues where staff spend a lot of time completing change requests only for the portal to briefly claim that there are errors on the form before it gets stuck in a state where the 3 dots appear to be loading and but doing anything quickly. Sometimes this is because of an attachment and the portal recovers an hour or so later or it is around the time that the portal timeout countdown begins.

Each time the information available in the logs is almost non-existent. I just wondered if any other companies have suffered from this, what they managed to find out and ultimately if they managed to find a workaround or solution?

Tony_Collett

I haven't heard of something like this, however you could have related item issues if users don't have the correct permissions to view or modify the related items. 

This could be a case of having to raise a support ticket in the Cireson Support Portal for more specific support with our team. 

Sean_Terry

Thanks. I have raised it - IR81310

Tom_Hendricks

You might want to rule out whether it is related to PR81151 (anti-forgery token error).  This is killing us right now, and to the user it just looks like the form is "thinking" when they try to submit it.  In reality, the error occurred right away and it is never going to submit.

Brian_Wiest

@Sean_Terry I suffered from this error for a long time. What I found it typically you will find an entry in the webconsole.log about the submission, it was killing me with change requests as well. After months of troubleshooting determined it was due to reviews in the change request template were failing to save. You might see something in the logs regarding “A discovery data item was rejected because the item is already bound to another Membership relationship.”

In my case it was due to migrating from SCSM 2012 to 2016 the LifeCycle tool has some bugs in coping the reviewers (fixed in later releases)

Does this sound close to what you have?

@Tom_Hendricks My findings on the anti-forgery token while it fills up the logs, it does not cause the three dots. It will slow down the users sessions but not work work stoppage. This will appear for orgs that have multi web servers on a load balancer. I cleared up most of the errors (still have rare pop up and I think due to timing of the auto logout sessions)

Cleared by One setting up sticky sessions. Two by making sure the "Machine Key" was the same on all IIS frontends

https://support.cireson.com/KnowledgeBase/View/2441#/

Now in the KB Cireson has you set the machine key at the Cireson portal level. While this works any upgrade you do will reset the value back to default. Since in my environment the web servers are dedicated to the SMP you can set at the IIS level.

HTH

Sean_Terry

Thanks @Tony_Collett, @Tom_Hendricks @Brian_Wiest. From what I've been told is that some staff open a new tab when their CR times out but now that little trick (to clear the timeout) doesn't work because it will issue a new token which doesn't match the original one. When the user tries to submit the system will reject it straight away but to the end user they will see the 3 dots and it looks like its trying to do something.

It seems like when you raise a CR it doesn't contact the server to show you are active so it can time out even as you type and trying to complete the ticket. It sounds to me like the portal should periodically contact the server to check if you are active and extend the timeout or refresh it. It should do this OOB.

Strangely we have following the KB when we upgraded the portal but we are still getting anti-forgery messages but as you said it might be the timeouts. Again, if you click refresh it should keep the original token or warn you that the session is dead and changes may be lost.

Tom_Hendricks

@Brian_Wiest, oh how I wish our situations were similar.  We have already spent a huge amount of time getting the machine keys to match, etc., but to no avail.  In our case, users do see the three dots every single time it happens.  There is a red 500 error from the server almost immediately, however.  If I go to the logs, there is always an anti forgery token error on the corresponding server at that time.

@Sean_Terry, in our case, it seems to be due to session timeout each time, since the keys definitely do match.  It sounds like we are similar in both regards.  When the session warning comes up and the user acknowledges it to keep their session going, that is apparently the point at which their token is no longer valid, but all indications on the screen are that everything is fine.  Then when the user tries to submit, they get the mismatched anti forgery token error.

It would in many cases be preferable for the user to get an "oops, you're about to lose an hour's worth of work, sorry that you have to ask a potentially very angry VIP all the same questions again!" error so that at least they could stop putting time into a form that has no chance of being saved, and could even copy it out to notepad possibly.  I have entered app.lib.mask.remove() on their browser consoles when people let me know about it in real time, to at least let them try to save their work out.  

As you pointed out, it would obviously be best if the system would either preserve the current token or properly establish a new one at the point the user keeps their session alive, though.

Sean_Terry

@Tony_Collett Is this something that Cireson can look at?

Happy to raise this as an idea if we can drum up enough votes. I don't think we could wait a year or two to reach a decent total though.

1) The warning should point out that refreshing the session will lose any changes
2) There should be an option in the pop-up to keep the changes if the user has made an error by leaving their console to long. Everyone gets distracted. Something similar to the Failure page. At least staff could recover their data.
3) Ideally, clicking refresh would take the info and open a new CR and populate what they had. It may be new but to the end user nothing has changed.

I'm sure there will be workarounds to this but it would be good if the portal did this out of the box.

Brian_Wiest

@Sean_Terry What do you have your session time out limits set to?

In testing a number of use cases after my timeouts occur, I found that after clicking an action there is a short moment before the Cireson script takes affect. So there was a need to make sure the various timers were set properly. My Example config,

Session State in IIS = 25 Minutes (Made sure set on all 5 portal servers)

SessionTimerCountdown = 300

UserTokenExpirationMinutes = 25

Sticky Session = 8 Hours

Sticky Mode Server Cookie or Source IP

Also made sure the machine keys were set to same value at IIS level on all 5 portal servers

With these settings once a user connects they will have a sticky session to a server for the day, with a 20 minutes session then a 5 minute count down. With the 5 minute count down it provides more time to click the refresh before it runs out.

After setting all these sessions the occurrence of the Anti-Forgery error significantly went down. I am now only seeing about 1 or 2 a day per portal server.

The two types I am seeing are

The required anti-forgery cookie "__RequestVerificationToken" is not present. - I expect this is users clearing their local cache. A standard practice from a number of users in the environment.

The anti-forgery cookie token and form field token do not match. - I expect this rare one is due to laptop users switching from hardware to wireless connections. Thus connecting to new portal server.

Does your webconsole.log show any errors during the save attempts?

Sean_Terry

Session State in IIS = 60 Minutes (Made sure set on both portal servers)

SessionTimerCountdown = 60

UserTokenExpirationMinutes = 60

Sticky Session = 8 Hours ??? Not sure how to check this?

Also made sure the machine keys were set to same value at IIS level on both portal servers

I would like to extend the countdown and implement the suggested code in IR81310

Brian_Wiest

If you have two portal servers you setup sticky sessions via your network load balancer

Recommend you up your sessiontimercountdown with it set at 60 they are only presented with a one minute count down. For me when they click the renew session while the 1 minute counter is running down it will perform the session renew would the need of the script from IR81310. Setting the session countdown to say 600 would allow them 10 minutes to click the renew button.

I have found that when working on any WI for a period of time opening another tab doesn't always keep that one page active. But clicking the renew button has worked in each case.

Of note after clicking the renew button there is a short delay before the renew session JS fires off. Since all are set to the same value you could have a case where analysts are hitting the renew at the last second but the actual renew command runs after the session is timed out. To help with that you might want to give a small bump token expiration.

Tony_Collett

Sean_Terry said:

@Tony_Collett Is this something that Cireson can look at?

Happy to raise this as an idea if we can drum up enough votes. I don't think we could wait a year or two to reach a decent total though.

1) The warning should point out that refreshing the session will lose any changes
2) There should be an option in the pop-up to keep the changes if the user has made an error by leaving their console to long. Everyone gets distracted. Something similar to the Failure page. At least staff could recover their data.
3) Ideally, clicking refresh would take the info and open a new CR and populate what they had. It may be new but to the end user nothing has changed.

I'm sure there will be workarounds to this but it would be good if the portal did this out of the box.

These are things that could be looked at if you were to contract Cireson Consulting services to create some modifications for you. Otherwise, the alternative is to post them on the Feature Requests board and hope they get implemented. 

Tom_Hendricks

As a customer, it is more than difficult to accept this as a feature request instead of a critical failure of the product that requires immediate focus of the highest priority.

Sure, anyone will concede that with the proper logical gymnastics all of the above can be categorized as "working as designed" but it is infuriating to users and they are just using the product normally but getting punished for it and they do not know why.  And most importantly, nothing can be done even by the most savvy user to avoid the loss of their work.

At least in our case (and there is an IR and a PR for it), users are powerless to keep their session alive no matter how attentive they are (no amount of lead time helps--clicking the warning message does not prevent the token from expiring even if the session stays alive).  They are essentially just getting punished for using the app too long and given false hope when they think they are keeping their session alive.  I have been told that sticky sessions would not necessarily solve this, as well.

Apart from being a terrible bullet point for the sales demo, this is simply not an acceptable user experience.  Data is lost, analyst work effort is doubled, customers are kept on the phone or called back to repeat themselves for no justifiable reason while losing extremely valuable billable time, and no workaround is known.  It needs to be better, and soon.  I look forward to someone invalidating this post with news of a fix.

seth_coussens

@Tom_Hendricks
I understand your frustration, and I hope you will reach out to me directly or @mention me in the future. I only recently heard of this particular issue...

clicking the warning message does not prevent the token from expiring even if the session stays alive

And I'm already looking into this to see if it's a configuration issue, confusion with documentation or out of the box settings, or a bug. I should have a better answer later this week.

Thanks,

Seth

Tom_Hendricks

I appreciate the response, and have full confidence in your team's ability to resolve it.  I'll @mention you in the future, no problem.