AWS reports that their services have been fully restored. This incident is now resolved.
Posted Nov 26, 2020 - 13:05 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Nov 25, 2020 - 21:22 UTC
Update
Old transcoding jobs have been reprocessed. There still a few failed transcoding jobs that we get requeue soon.
Posted Nov 25, 2020 - 21:17 UTC
Update
The transcoding service is slowly processing the old jobs, thanks for your patience.
Posted Nov 25, 2020 - 20:16 UTC
Update
We have been able to mitigate the issue. New uploaded media will now be transcoded normally but old media will not get reprocessed yet.
Posted Nov 25, 2020 - 19:20 UTC
Update
Related to the AWS outage, there is currently very little transcoding job processed. There around 2 hours of waiting time to get media transcoded and it's going up.
Posted Nov 25, 2020 - 18:23 UTC
Update
We are observing longer than usual delay before uploaded media are transcoded.
Posted Nov 25, 2020 - 17:24 UTC
Update
We have now observed a an impact on the Shotgun notification service. AWS has acknowledged the issue and are working towards a solution.
Posted Nov 25, 2020 - 15:17 UTC
Investigating
We're noticing some upstream issue in certain AWS services. Currently Shotgun is not impacted, but we're actively monitoring the situation.
Posted Nov 25, 2020 - 14:24 UTC
This incident affected: Transcoding Service and Notification Service.