Service Degradation - Shotgun
Incident Report for Flow Production Tracking
Postmortem

What happened?

On Thursday April 7th 2021, from 14:05UTC to 16:39UTC, the Inbox app was unavailable. During the rolling deployment of a minor version of Shotgun, a change-related defect resulted in the Inbox app being unavailable for clients whose site was being updated. As soon as we discovered the issue we initiated a rollback to the previous version of Shotgun, which resolved the issue for all sites.

Scope of impact

During the incident, the Inbox app was unavailable. Users that attempted to access their Inbox would see an error message stating, "App Not Available." No other functionality within Shotgun was affected and no Inbox messages were lost.

What we'll do to prevent this incident from happening again?

A new process to protect against this type of change-induced issue has been put in place. This process will allow our development team to continue to deliver updates and features, while removing the risk of this type of error happening during a deployment of Shotgun.

Posted Apr 13, 2021 - 20:24 UTC

Resolved
This incident has been resolved.
Posted Apr 07, 2021 - 15:41 UTC
Identified
We have identified an issue with this morning's deployment. Shotgun Inbox is currently unavailable for some users. A fix is being implemented.
Posted Apr 07, 2021 - 14:30 UTC
Investigating
We are observing a high number of failed requests to the Shotgun service which may impact site availability for some clients. Email notifications may also be delayed. This issue is under investigation.
Posted Apr 07, 2021 - 14:14 UTC
This incident affected: Flow Production Tracking and Notification Service.