SWE-Milestone

Evaluating AI agents on continuous software evolution, structured as a milestone DAG, across real release histories.

Paper ↗ GitHub ↗ Dataset ↗ Trajectories ↗

Pareto frontier

Leaderboard

Notes

Time is wall-clock, so it varies with each agent's system prompt and with API response latency. Treat it as a rough indicator rather than an exact efficiency measure.
Unless otherwise noted, Claude Code runs use a 200K context window.

About SWE-Milestone

SWE-Milestone evaluates what isolated-patch benchmarks cannot: whether an agent can carry a codebase forward. Each task instance is a real open-source repository and a range of consecutive releases. We reconstruct the development history between those releases into an executable milestone DAG, where every milestone carries its own verifiable checks.

What does SWE-Milestone evaluate?

Continuous software evolution. The agent must carry a real codebase through a full range of releases in one unbroken session, not produce isolated patches.

How is a milestone scored?

Each milestone is scored on a precision–recall structure over its own tests: recall credits the fail-to-pass tests (F2P) the agent gets passing, precision credits the previously-passing tests (P2P) it keeps intact. A milestone counts as fully resolved only when every F2P passes and no P2P regresses, penalizing both under-building and regressions.

How is this different from isolated-patch benchmarks?

There is no reset between milestones. Earlier mistakes, context pollution and drifting instructions compound across the run, exactly as they do in real long-horizon engineering. That is precisely the failure mode isolated-patch settings cannot observe.

Tasks

Each task is one repository over a chosen release range, reconstructed into a milestone DAG where every milestone carries its own tests. The agent works through the whole DAG in a single continuous run, implementing one milestone at a time; a milestone only unlocks once the milestones it depends on are complete.

zeromicro/go-zero ··· 110K LoC Go

v1.6.0→v1.9.3 49 releases 466 commits

Advances the goctl code generator and API/proto tooling, adds runtime features, and lands broad bug fixes and dependency upgrades.

+4,462−1,941 23 Milestones

element-hq/element-web ··· 476K LoC TypeScript

v1.11.95→v1.11.97 4 releases 75 commits

Introduces the new room list with its view-model, store and filtering, improves group-call error handling and accessibility, and fixes assorted UI issues.

+4,999−2,648 18 Milestones

nushell/nushell ··· 264K LoC Rust

0.106.0→0.108.0 2 releases 199 commits

Refines the parser and custom completions, adds shell features such as debounced file watching, and fixes many panics and regressions.

+9,717−5,803 13 Milestones

apache/dubbo ··· 350K LoC Java

3.3.3→3.3.6 3 releases 134 commits

Improves server-streaming RPC and configuration handling, removes deprecated test modules, and lands a large batch of bug fixes and dependency upgrades.

+3,429−725 12 Milestones

scikit-learn/scikit-learn ··· 280K LoC Python

1.5.2→1.6.0 2 releases 311 commits

Expands Array API support and metadata routing across estimators and metrics, fixes sparse-array validation, and brings extensive documentation and test improvements.

+4,572−2,800 12 Milestones

BurntSushi/ripgrep ··· 48K LoC Rust

14.1.1→15.0.0 1 release 95 commits

Adds file-type detection for many new languages, extends and hardens the glob engine, refines replace and JSON output, and migrates to the Rust 2024 edition.

+1,080−394 11 Milestones

navidrome/navidrome ··· 144K LoC Go

v0.57.0→v0.58.0 1 release 35 commits

Hardens the plugin system with metrics and error handling, fixes the scanner's cover-art and lyrics parsing, and addresses several UI and Subsonic issues.

+4,695−1,205 9 Milestones

Analysis

How the top-15 models' scores change across normalized milestone execution progress and milestone complexity, and how Precision and Recall accumulate through each run. Milestone execution progress bins weight every repository equally.

Score over Milestone Execution Progress

Models · click to show/hide

Accumulated Recall & Precision over Milestone Execution Progress

RecallPrecisionIdeal

Select models to compare

Citation

@misc{deng2026swemilestoneevaluatingaiagents,
      title={SWE-Milestone: Evaluating AI Agents on Continuous Software Evolution},
      author={Gangda Deng and Zhaoling Chen and Zhongming Yu and Haoyang Fan and Yuhong Liu and Yuxin Yang and Dhruv Parikh and Rajgopal Kannan and Le Cong and Mengdi Wang and Qian Zhang and Viktor Prasanna and Xiangru Tang and Xingyao Wang},
      year={2026},
      eprint={2603.13428},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2603.13428},
}