Repository navigation
CI metrics — main — 2026-09-30 to 2026-10-06 UTC #2764
Description
CI metrics — main — 2026-09-30–2026-10-06 UTC
Summary
- Repeated six-hour limits dominated observed PR test failures.
Week-over-week
Rates (0–100%)
| Metric | Previous week | This week |
|---|---|---|
| PR commits rerun | ██ 14.0% (6/43) | █ 11.8% (18/153) |
| Rerun rate in post-first-rerun PR cohort | █████████ 75.0% (6/8) | ████████ 64.3% (18/28) |
| Retried PR commits recovered | ████████ 66.7% (4/6) | ███████ 61.1% (11/18) |
| Nightly cycles failed | ███ 28.6% (2/7) | █████████ 71.4% (5/7) |
Hours (bars share a scale within each row)
| Metric | Previous week | This week |
|---|---|---|
| Ready for review → all approvals (mean) | ████████████ 280.966 h (n=2) | ████ 89.743 h (n=19) |
| /merge → merge (mean) | ██████ 6.489 h (n=5) | ████████████ 12.841 h (n=21) |
| PR initial e2e pass (mean) | ████████████ 3.898 h (n=10) | ███████ 2.145 h (n=23) |
| Nightly end-to-end CI time (mean) | ████████████ 4.389 h (n=5) | ██████████ 3.563 h (n=1) |
| Nightly observed broken time | ████████ 61.2 h | ████████████ 87.5 h |
PR review and merge latency
| Metric | Samples | Min (h) | Mean (h) | Max (h) | Std dev (h) | Prior mean (h) |
|---|---|---|---|---|---|---|
| Ready for review → all approvals | 19 | 0.038 | 89.743 | 617.073 | 158.849 | 280.966 (n=2) |
| /merge → merge | 21 | 0.002 | 12.841 | 214.488 | 45.607 | 6.489 (n=5) |
21 merged PRs; approval completion unresolved for 2; ready-for-review start unavailable for 0; /merge absent for 0.
PR CI
Reruns: 18/153 (11.8%) distinct commits; 21 additional attempts (15 commits ×1, 3 commits ×2); 11/18 (61.1%) retried commits recovered green.
| Metric | Samples | Min (count) | Mean (count) | Max (count) | Std dev (count) | Prior mean (count) |
|---|---|---|---|---|---|---|
| Retries per observed commit | 153 | 0.0 | 0.1 | 2.0 | 0.4 | 0.2 (n=43) |
| Failed job executions per retried commit | 18 | 0.0 | 5.8 | 34.0 | 7.7 | 2.3 (n=6) |
Failed jobs include original/retry executions and downstream PR gates.
Complete successful PR passes
| Metric | Samples | Min (h) | Mean (h) | Max (h) | Std dev (h) | Prior mean (h) |
|---|---|---|---|---|---|---|
| Initial e2e CI pass | 23 | 0.603 | 2.145 | 5.682 | 1.310 | 3.898 (n=10) |
Failure gist
Failure exposure: 104/3082 (3.4%) executed jobs. product/build/test: 4; external infrastructure: 1; unknown: 99.
- Required-job gate reported downstream failures — 25 failed jobs; 15 PRs; 11 recovered; unknown (high confidence). The gate listed failed required jobs; linked jobs carry the underlying symptoms. Evidence: job 1, job 2.
- Conda C++ builds lacked an RMM header — 4 failed jobs; 1 PR; 0 recovered; product/build/test (high confidence). Four PR Add test parameter space guidance to cpp REVIEW_GUIDELINES #2711 matrix jobs failed compiling raft because rmm/cuda_stream_view.hpp was absent. Evidence: job 1, job 2.
- V100 Rocky C++ test jobs hit six-hour limit — 4 failed jobs; 4 PRs; 1 recovered; unknown (high confidence). The check annotations confirm six-hour execution limits; the underlying delay is unknown. Frequent configuration: 12.2.2, 3.11, amd64, rockylinux8, v100, earliest-driver, oldest-deps (4 jobs). Evidence: job 1, job 2.
- H100 Ubuntu C++ test jobs hit six-hour limit — 4 failed jobs; 3 PRs; 0 recovered; unknown (high confidence). The check annotations confirm six-hour execution limits; the underlying delay is unknown. Frequent configuration: 12.9.2, 3.14, amd64, ubuntu22.04, h100, latest-driver, latest-deps (4 jobs). Evidence: job 1, job 2.
- L4 Rocky C++ test jobs hit six-hour limit — 4 failed jobs; 3 PRs; 0 recovered; unknown (high confidence). The check annotations confirm six-hour execution limits; the underlying delay is unknown. Frequent configuration: 13.0.3, 3.14, arm64, rockylinux8, l4, latest-driver, latest-deps (4 jobs). Evidence: job 1, job 2.
- Other: 63 distinct failed jobs across 36 smaller groups; 0 also appear above.
25 distinct failed jobs later passed on a comparable retry (flake candidates).
Nightly CI
Cycles: 7 observed; 6 paired; 1 unpaired; 0 ambiguous. Failed: 5/7 (71.4%); retried: 1/7 (14.3%) cycles / 1 attempt.
| Metric | Samples | Min (count) | Mean (count) | Max (count) | Std dev (count) | Prior mean (count) |
|---|---|---|---|---|---|---|
| Retries per cycle | 7 | 0.0 | 0.1 | 1.0 | 0.3 | 0.1 (n=7) |
| Failed jobs per cycle | 7 | 0.0 | 4.6 | 14.0 | 5.4 | 0.4 (n=7) |
Complete successful nightly passes
| Metric | Samples | Min (h) | Mean (h) | Max (h) | Std dev (h) | Prior mean (h) |
|---|---|---|---|---|---|---|
| Nightly end-to-end CI time | 1 | 3.563 | 3.563 | 3.563 | 0.000 | 4.389 (n=5) |
Failure gist
Failure exposure: 32/890 (3.6%) executed jobs. product/build/test: 2; external infrastructure: 0; unknown: 31.
- H100 nightly Python tests hit six-hour limit — 2 failed jobs; 2 cycles; 0 recovered; unknown (high confidence). Two nightly cycles had H100 conda Python test jobs canceled at the confirmed six-hour limit. Frequent configuration: 12.9.2, 3.13, amd64, ubuntu24.04, h100, latest-driver, latest-deps (2 jobs). Evidence: job 1, job 2.
- H100 nightly wheel tests hit six-hour limit — 2 failed jobs; 2 cycles; 0 recovered; unknown (high confidence). Two nightly cycles had H100 wheel test jobs canceled at the confirmed six-hour limit. Frequent configuration: 12.9.2, 3.12, amd64, ubuntu24.04, h100, latest-driver, latest-deps (2 jobs). Evidence: job 1, job 2.
- GH200 IVF-RABITQ test failed — 1 failed job; 1 cycle; 0 recovered; product/build/test (medium confidence). IvfRabitq build_host_input_search/1 failed in the GH200 C++ test matrix; the cause is unclear. Evidence: job 1.
- GB300 fused-neighbor test case 34 failed — 1 failed job; 1 cycle; 0 recovered; product/build/test (medium confidence). NNTest_fp32_fused.test/34 failed in the October 6 GB300 C++ test job; the cause is unclear. Evidence: job 1.
- GB300 fused-neighbor test case 35 failed — 1 failed job; 1 cycle; 0 recovered; product/build/test (medium confidence). NNTest_fp32_fused.test/35 failed in that same job; the cause is unclear. Evidence: job 1.
- Other: 27 distinct failed jobs across 38 smaller groups; 1 also appear above.
12 distinct failed jobs later passed on a comparable retry (flake candidates).
Nightly broken periods
3 observed incidents; 87.5 broken hours inside this window; 1 unresolved at its end; 0 incidents lack a prior observed green baseline.
| Metric | Samples | Min (h) | Mean (h) | Max (h) | Std dev (h) | Prior mean (h) |
|---|---|---|---|---|---|---|
| Completed incident duration | 2 | 6.295 | 35.160 | 64.025 | 28.865 | 290.766 (n=2) |
Completed durations may span report boundaries.
Open incidents: since 2026-10-06T06:47:47Z (17.2 hours at window end).
- 2026-09-30: 64.0 broken hours in this window. GH200 IVF-RABITQ test failure began the incident; later H100 test jobs hit six-hour limits. No shared cause is established. Failure evidence. Recovered: stage 1, stage 2.
- 2026-10-05: 6.3 broken hours in this window. Python test jobs failed across A100 and H100 variants; the excerpts do not identify one cause. The cycle later recovered. Failure evidence. Recovered: stage 1, stage 2.
- 2026-10-06: 17.2 broken hours in this window. GB300 fused-neighbor tests failed, alongside distinct H100 test failures; no green recovery was observed by week end. Failure evidence. Still unresolved at window end.
Suggested actions
- Inspect the missing rmm/cuda_stream_view.hpp header in PR Add test parameter space guidance to cpp REVIEW_GUIDELINES #2711 conda builds. (evidence)
- Investigate repeated six-hour timeouts in the listed GPU test matrices. (evidence)
- Review the GB300 fused-neighbor failures in the unresolved October 6 nightly. (evidence)
Coverage
Window: [2026-09-30T00:00:00Z, 2026-10-07T00:00:00Z); collected 2026-10-07T01:45:40Z. PR context since 2026-09-23T00:00:00Z; nightly context since 2026-08-01T00:00:00Z. 198 PR runs and 105 nightly stage runs inspected; 0 failed or setup job logs unavailable; 0 dispatched runs unclassified; 6 current-window runs unfinished at collection; 109 ordinary branch builds excluded across the collected context. Std dev is population-based; prior means include sample counts. Full-pass timing uses initial triggers only; intentional skips count. Reruns belong to retry metrics. Post-first-observed-rerun cohort: 28/153 observed PR+SHA pairs; inflection commit included; 5/17 cohort PRs predate scan. Counts are CI observations, not total pushes. Failure groups can overlap; rates show their denominators. Missing nightly runs are coverage gaps. Approval-completion estimates rely on current team membership where historical membership is unavailable; post-approval commits are treated as resetting approvals; merge-command eligibility uses current collaborator permissions.
Activity