John Yang

@jyangballin

CS PhD @Stanford. Created @SWEbench (multi-lingual/modal); SWE-agent; SWE-smith; SWE-sweep; InterCode; CodeClash; ProgramBench

SF Bay Area

john-b-yang.github.io

Joined June 2012

1,131 Following

7,558 Followers

Replies Reposts Media

Pinned
John Yang
@jyangballin
May 5
How much of SQLite, FFmpeg, PHP compiler can LMs code from scratch? Given just an executable and no starter code or internet access. Introducing ProgramBench: 200 rigorous, whole-repo generation tasks where models design, build, and ship a working program end to end. 🧵
Kilian Lieret
109
John Yang
@jyangballin
Oct 29
Another great whole-repo-generation benchmark led by fantastic @HantianDing! My main learning from Hantian's work - there's a lot of creative ways to go about getting a model to build out a repo from scratch. ProgramBench gives an executable. E2E-SWE gives a specification.
Hantian Ding
@HantianDing
Oct 29
Introducing E2E-SWE, a benchmark for reliably measuring how well coding agents turn a specification into a working software repository. Agents start from an empty workspace, work offline, and succeed only when their project installs and passes every hidden functional test.
4
John Yang
@jyangballin
Oct 28
SWE-sweep on the Gemini 4 Carbon model card! Congrats @OfficialLoganK @Google and the Gemini team on the fantastic numbers! Follow @KLieret for leaderboard updates, a lot of new numbers coming in :D
Logan Kilpatrick
@OfficialLoganK
Oct 28
Introducing Gemini 4 Carbon, it's a big jump over Argon which we released just 2 weeks ago. Gemini 4 Carbon is SOTA on cyber, long horizon tasks, ranks #1 on Artificial Analysis V5, significantly improves on usability in domains like coding, and is more aligned.
1
John Yang
@jyangballin
Oct 28
SWE-sweep, SWEeper-bench, now PROBE. Many coding benchmarks in the past year have gone in the direction of proposing harder versions of tasks w/ fundamentally the same SWE-bench style formulation. In comparison, proactivity is a great direction - much less about making things
Mark Müller
@mnmueller
Oct 28
New paper: Frontier Coding Agents Find Different Bugs Than Maintainers Fix In proactive bug finding, agents report fewer than 20% of the bugs humans fixed. But they report a lot of other bugs, most of which are valid. This points towards an important distributional mismatch.
4
John Yang
@jyangballin
Oct 23
Saturating a benchmark that tests models on rebuilding ffmpeg is gonna take a fraction of the 3 yrs it took to saturate GitHub issue resolution. Crazy times we live in.
5

John Yang

John Yang’s posts