Postmortem: Missing task logs and traces

Date: 2026-09-03
Duration: ~67 minutes
Impact: New logs, spans and trace data for some runs were not stored and cannot be recovered

Summary

Between approximately 14:35 and 15:43 UTC on 2026-09-03, batches of new task events (the logs, spans and span events that make up a run's trace) were rejected by ClickHouse, the database behind the trace view, logs view and event search. Each batch was retried three times over about 1.5 seconds and then dro...