Scientific ML Agent Benchmark

NatureBench

Can coding agents match the published SOTA of Nature-family papers?

We evaluate frontier coding-agent configurations on 90 Nature-family scientific ML tasks in isolated containers.

arXiv GitHub Hugging Face

Leaderboard

Evaluation Setup

Reported runs
Per-task timeout 4 hours

Agent solve budget

Compute One GPU per GPU task

gpu_low: NVIDIA RTX 3090/4090 (24 GB); gpu_high: NVIDIA A800 (80 GB, A100-class); cpu: no GPU.

View task-group lists ↗
Web search Disabled

For all evaluated agents

Metric Definitions

g
SOTA-normalized relative gap, g = dir * (m - m_sota) / |m_sota|; dir handles metric direction.
Surpass-SOTA
Share of judge-valid tasks with g > 0.1.
Match-SOTA
Share of judge-valid tasks with g >= 0; g = 0 matches paper SOTA.
CR
Completion Rate, the valid-score rate.
SR
Score Rate, the any-score rate.
90
Tasks
6
Scientific domains
15.6%
Best Surpass-SOTA
41.1%
Best Match-SOTA
88.9%
Top completion

Main Leaderboard

Numeric ranking of coding-agent configurations by Surpass-SOTA.

Rank Model Harness Surpass-SOTA Match-SOTA Median g (all) CR SR Invalid

Score Distribution

Task-level g bins per configuration

Domain Breakdown

Domain labels follow the six-domain taxonomy used in the NatureBench paper. Domain rankings use the same Surpass-SOTA rule as the overall leaderboard.

Domain Ranking

Domain Winners

Best configuration within each scientific domain

Domain N Winner Surpass-SOTA Match-SOTA Median g (all)

Case-Level View

Each cell shows task-level normalized relative gap g. Values above 0.1 count as Surpass-SOTA; nonnegative values match the paper's reported SOTA.

Surpass-SOTA, g > 0.1 Match-SOTA, 0 <= g <= 0.1 Below SOTA, g < 0 Invalid No score/submission Outlined cell: best configuration for that task

Per-Case Score Matrix

Submit Results

We welcome community submissions to the NatureBench leaderboard.

Cite NatureBench

If you use NatureBench in your research, please cite our work.

BibTeX 
@misc{wang2026naturebench,
  title         = {NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?},
  author        = {Yuru Wang and Lejun Cheng and Yuxin Zuo and Sihang Zeng and Bingxiang He and Che Jiang and Junlin Yang and Yuchong Wang and Kaikai Zhao and Weifeng Huang and Kai Tian and Zhenzhao Yuan and Jincheng Zhong and Weizhi Wang and Ning Ding and Bowen Zhou and Kaiyan Zhang},
  year          = {2026},
  eprint        = {2606.24530},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2606.24530}
}