Resolved -
Build times are normal for all new builds. Background queues have been cleared of 99% of backlogged jobs; the only ones that remain are for large repos (>5K source files), which should clear in the next 30-45 minutes depending on size.
No further effects on latency expected.
Closing this incident.
Dec 18, 08:58 PST
Update -
Update:
- Normal build times have been recovered for all new jobs.
- Backlogged jobs are 80% cleared (20% remaining).
We will continue clearing backlogged jobs until cleared and monitoring for any further issues.
Dec 18, 07:31 PST
Monitoring -
We had an incident overnight that entailed several server outages affecting APAC and EU users. The impact has been latency in recent build times for those users. We have resolved the issue and are scaling to clear backlogged background jobs. US Central and West users should not be affected.
Dec 18, 06:51 PST