Service Degradation - Shotgun
Incident Report for Flow Production Tracking
Postmortem

On Thursday July 30th 2020, from 15h32 to 16h55 UTC, some sites were incorrectly suspended. Affected clients were unable to access their site for a period during the incident.

What happened?

During a minor housekeeping operation a number of Shotgun sites were erroneously suspended for violation of payment terms. All affected sites were subsequently restored as soon as the oversight was identified.

Scope of impact

A limited number of sites were unavailable during the incident. Affected clients were unable to access their site until restored by Shotgun.

What will be done to prevent this incident from happening again?

We have made improvements to our internal administration API to safeguard against inappropriate changes being made accidentally in the future.

Posted Aug 03, 2020 - 17:47 UTC

Resolved
This incident has been resolved.
Posted Jul 30, 2020 - 19:22 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 30, 2020 - 17:00 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Jul 30, 2020 - 16:39 UTC
Investigating
We are observing a high number of failed requests to the Shotgun service which may impact site availability for some clients. Email notifications may also be delayed. This issue is under investigation.
Posted Jul 30, 2020 - 16:04 UTC
This incident affected: Flow Production Tracking and Notification Service.