What happened?

On Friday May 21, 2021 from 20:26UTC to 22:36UTC the ShotGrid transcoding service experienced an outage which resulted in failed transcodes of uploaded Versions, along with other non-Version based thumbnails including Files, Note attachments and annotations.

The cause of the issue was a codepath silently failing under a rare set of circumstances.

Scope of impact

The incident affected all hosted ShotGrid sites. During the outage window, successful uploads to ShotGrid were not transcoded and associated thumbnails were unavailable. The unusual nature of the failure extended the types of media impacted beyond Versions, also affecting Files and Note attachments and annotations.

No media was lost and all media uploaded to ShotGrid during the incident was saved in the respective storage location for each ShotGrid site until it could be successfully transcoded once the transcoding service recovered.

What we'll do to prevent this incident from happening again?

Following our investigation into the root cause we'll be taking a number of measures to detect and prevent this type of outage from recurring in the future.

• We'll be reviewing the code for our transcoding service to ensure the codepath that lead to the incident is updated to prevent similar issues
• We're updating our monitoring tooling to ensure this type of outage is caught earlier