Users can't run 2013 workflows in SharePoint Online

Incident
August 19, 2:00pm

Users can't run 2013 workflows in SharePoint Online

Status: closed
Start: August 19, 9:05am
End: August 19, 2:00pm
Duration: 4 hours 54 minutes
Affected Components:
SharePoint (The Hub)
Update

August 19, 9:05am

August 19, 9:05am

Title: Some users can't run 2013 workflows in SharePoint Online
User Impact: Users are unable to run 2013 workflows in SharePoint Online.
Current status: We’re reviewing details provided by affected users to determine the next steps needed to resolve this issue.
Scope of impact: Users who are served through the affected infrastructure in North America and Canada may experience impact.
Next update by: Thursday, August 19, 2021, at 4:00 PM UTC

Update

August 19, 10:40am

August 19, 10:40am

Current status: We’re reviewing back-end service logs to better understand the issue and determine our next troubleshooting steps.
Scope of impact: Users who are served through the affected infrastructure in North America and Canada may experience impact.
Next update by: Thursday, August 19, 2021, at 6:00 PM UTC

Update

August 19, 12:08pm

August 19, 12:08pm

Current status: We've noticed some errors during our review, leading us to suspect that a networking issue may be the cause. We're reviewing the API version of the service, as well as performing connectivity tests to isolate the cause of impact and determine our next steps.
Scope of impact: Users who are served through the affected infrastructure in North America and Canada may experience impact.
Next update by: Thursday, August 19, 2021, at 8:00 PM UTC

Resolved

August 19, 2:00pm

August 19, 2:00pm

Final status: We've completed restarting the infrastructure, and after reviewing workflow telemetry and reaching out to affected users, we’ve confirmed that impact has been fully remediated.
Scope of impact: Users who were served through the affected infrastructure in North America and Canada may have experienced impact.
Start time: Thursday, August 19, 2021, at 6:50 AM UTC
End time: Thursday, August 19, 2021, at 7:00 PM UTC
Root cause: A portion of infrastructure that supports workflow calls unexpectedly stopped performing within acceptable standards, causing impact.
Next steps: - We're reviewing this portion of infrastructure to understand why it unexpectedly stopped performing within acceptable standards to help prevent similar impact from reoccurring in the future. - We're reviewing our monitoring services to look for ways to reduce detection time and more quickly restore service.
This is the final update for the event.

Resolved

August 19, 2:00pm

August 19, 2:00pm

Current status: We've confirmed that a portion of infrastructure that supports workflow calls unexpectedly stopped performing within acceptable standards, causing impact. We've re-routed traffic to an alternate infrastructure and are restarting it, after which we'll review telemetry and reach out to affected users to confirm impact remediation.
Scope of impact: Users who are served through the affected infrastructure in North America and Canada may experience impact.
Start time: Thursday, August 19, 2021, at 6:50 AM UTC
Root cause: A portion of infrastructure that supports workflow calls unexpectedly stopped performing within acceptable standards, causing impact.
Next update by: Thursday, August 19, 2021, at 9:30 PM UTC