We are no longer detecting the issues as of the rollback, and we are marking the incident as resolved. Our team will continue to investigate the underlying cause to make sure similar situations are avoided in the future. We are adding additional monitoring systems and test suites to better detect anomalies before wider-scale deployments.
Posted May 03, 2023 - 17:57 UTC
Monitoring
The rollback has now been fully applied globally. We are no longer observing any issues. We are now monitoring the situation.
Posted May 03, 2023 - 16:27 UTC
Identified
We are experiencing a rare issue affecting a small number of requests to large files with the "Optimize for Video Delivery" enabled, where the requests will time out in the middle of the download. There is no widespread impact on the accessibility of files, and no traffic reduction has been recorded. Less than 0.01% of requests have been affected, according to our internal logging.
Our team is working on resolving the issue urgently, and the rollback has already been initiated in most regions around the world, with an ETA resolution time of one hour.
Background: Starting at approximately April 29th, over a period of a couple of days, we began rolling out a global update to our CDN infrastructure to the latest Nginx software version alongside a series of performance and reliability patches. The update has prior been slowly tested and rolled out over a period of over two months in multiple regions around the world to assure system stability. There was no recorded impact during this time due to the very rare occurrence of the issue. During the wider deployment, we began receiving reports of the issue, which was later tied to this update deployment.