John Yang (@jyangballin) / X

John Yang

886 posts

John Yang

@jyangballin

CS PhD @Stanford. Created @SWEbench (multi-lingual/modal); SWE-agent; SWE-smith; InterCode; CodeClash; ProgramBench

SF Bay Area

john-b-yang.github.io

Joined June 2012

Pinned
John Yang
@jyangballin
May 5
How much of SQLite, FFmpeg, PHP compiler can LMs code from scratch? Given just an executable and no starter code or internet access. Introducing ProgramBench: 200 rigorous, whole-repo generation tasks where models design, build, and ship a working program end to end. 🧵
John Yang
@jyangballin
Apr 2, 2024
SWE-agent is our new system for autonomously solving issues in GitHub repos. It gets similar accuracy to Devin on SWE-bench, takes 93 seconds on avg + it's open source! We designed a new agent-computer interface to make it easy for GPT-4 to edit+run code github.com/princeton-nlp/…
John Yang
@jyangballin
May 7, 2025
40% with just 1 try per task: SWE-agent-LM-32B is the new #1 open source model on SWE-bench Verified. We built it by synthesizing a ton of agentic training data from 100+ Python repos. Today we’re open-sourcing the toolkit that made it happen: SWE-smith.
John Yang
@jyangballin
Nov 5, 2025
New eval! Code duels for LMs ⚔️ Current evals test LMs on *tasks*: "fix this bug," "write a test" But we code to achieve *goals*: maximize revenue, cut costs, win users Meet CodeClash: LMs compete via their codebases across multi-round tournaments to achieve high-level goals
00:00
John Yang
@jyangballin
May 6, 2024
The SWE-agent preprint has finally landed! Check it out at swe-agent.com/paper.pdf
John Yang
@jyangballin
Sep 29, 2025
"Your fancy AI scaffolds will be washed away by scale"⁣ - Noam Brown If you're still doing SWE-agent research with heavy frameworks + manually defined tools, don't. All those artifacts (exec. traces, tools, prompts) don't transfer. Use mini-SWE-agent. Bash-only is the future.
Kilian Lieret
@KLieret
Sep 29, 2025
We evaluated Anthropic's Sonnet 4.5 with our minimal agent. New record on SWE-bench verified: 70.6%! Same price/token as Sonnet 4, but takes more steps, ending up being more expensive. Cost analysis details & link to full trajectories in 🧵
John Yang
@jyangballin
Apr 2, 2024
Replying to @jyangballin
Simply connecting an LM to a vanilla bash terminal does not work well. Our key insight is that LMs require carefully designed agent-computer interfaces (similar to how humans like good UI design) E.g. When the LM messes up indentation, our editor prevents it and gives feedback
John Yang
@jyangballin
Oct 7, 2024
We're launching SWE-bench Multimodal to eval agents' ability to solve visual GitHub issues. - 617 *brand new* tasks from 17 JavaScript repos - Each task has an image! Existing agents struggle here! We present SWE-agent Multimodal to remedy some issues Led w/ @_carlosejimenez 🧵
John Yang
@jyangballin
Jun 3, 2024
We just updated the SWE-agent preprint: - New results: 87.7% Pass@1 on HumanEvalFix (SoTA!) - Full SWE-agent walkthrough + ACI definition - More analysis (Failure modes, pass@k, agent behaviors, + ton more) ACI is 🔑 to LMs as SWEs, here: arxiv.org/abs/2405.15793 1/3
arxiv.org
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Language model (LM) agents are increasingly being used to automate complicated tasks in digital environments. Just as humans benefit from powerful software applications, such as integrated...
John Yang
@jyangballin
Oct 30, 2025
man I'm excited, can't wait to reveal soon what @KLieret @_carlosejimenez @OfirPress @lschmidt3 @Diyi_Yang and I have been working on. Where should code evaluation go after SWE-bench? We have an answer ⚔️
John Yang
@jyangballin
Jan 16, 2025
SWE-bench Multimodal evaluation code is out now! SWE-bench MM is a new set of JavaScript issues that have a visual component (‘map isn’t rendering correctly’, ‘button text isn’t appearing’).
John Yang
@jyangballin
Apr 2, 2024
Replying to @jyangballin
Another example is that we discovered that for viewing files, letting SWE-agent only view 100 lines at a time was better than letting it view 200 or 300 lines and much better than letting it view the entire file. Good agent-computer design is important even when using GPT-4. 4/N
John Yang
@jyangballin
Oct 22, 2024
📊 SWE-bench Leaderboard Update! swebench.com SWE-bench Lite 🥇 Bytedance MarsCode Agent 🥈 Honeycomb 🥉 Gru (2024-08-11) SWE-bench Verified 🥇 [Anthropic] Tools + Claude 3.5 Sonnet (2024-10-22) 🥈 Solver (2024-09-12) 🥉 Gru(2024-08-24) Keep reading for summaries!
John Yang
@jyangballin
May 4, 2025
@ weekend warriors - DM me a GitHub repo that you like / maintain, and I'll train you a 7B coding agent that's an expert for that repo. Main constraints - it's predominantly Python, and has a testing suite w/ good coverage. (example of good repo = sympy, pandas, sqlfluff)