Streaming live at 10am (PST)

[Incident Report] Editor Inaccessible



On Monday at 7:40am PST (01/02/2018), we deployed a faulty build that caused the Editor to be inaccessible. Users attempting to load the editor either directly via the ?edit URL or from the dashboard saw a blank page. At 1:04pm PST we had a new deploy that restored the Editor for our customers. The length of time to revert was due to our engineering team not being aware of the issue for a period of time and issues with our continuous integration service that delayed restoring a previous build. The total amount of time the Editor was inaccessible was around 5 hours and 24 minutes or less depending on user cache hits.

Next Steps

We will be taking steps to normalize our build configurations to detect issues sooner, and implementing more sanity checks for the Editor to prevent faulty builds in production. Additionally we will be improving our ability to rollback deploys in a timely manner.

We apologize about this incident and are working to prevent this from happening again in the future.


Should also update here, that way I can use the RSS feed to notify our Slack channel of downtimes.