Pinned
The first ProgramBench task was just solved by GPT 5.5 high/xhigh. Interestingly, high/xhigh picked two different languages for the task (C vs Python). GPT 5.5 xhigh was significantly better than Opus 4.7 xhigh in all metrics. š§µ
Kilian Lieret
@KLieret
Meta Superintelligence, prev. Princeton. SWE-bench multilingual/multimodal, SWE-agent, mini-swe-agent, SWE-smith, CodeClash, ProgramBench
- mini-swe-agent with Opus 5 is significantly ahead on TerminalBench 3/FrontierBench. Interestingly it even beats Fable 5 + Claude Code.
- Had a great time talking at @ScaleAILabs. New levels of model capabilities require new benchmarking paradigms. Also really enjoyed talks by @sirrice @mohit_r9a and @MiguelR33478246! Thanks to all the organizers and to @liuying04
- Extremely cool project: What's the best SWE-bench score you can achieve by training a model from scratch with limited $ budget. SWE-bench might be near saturated for frontier models, but it still has a lot of life in it for hillclimbing with smaller or from scratch modelsWe turned @karpathy's nanochat into a SWE-bench speedrun! ā” $60 of train compute ā 5.0% pass@1 ā Claude 2 (2023 SOTA) ā” $1000 ā 11.0% pass@1 > Claude 3 Haiku From randomly initialized model weights! How? š§µš
- When SWE-bench was launched initially, people questioned whether it was feasible at all. We got very similar feedback for ProgramBench. Up until now, only 2/200 tasks were solved. But Opus 5 just made an enormous jump with 9 fully solved tasks instances!Takeoff fully in motion: Claude Opus 5 (xhigh) is the new #1 on ProgramBench, and it's not close. ProgramBench asks a coding agent to rebuild a whole program (sqlite, ffmpeg, php) from scratch. Previous high: GPT 5.6 Sol w/ 2 Opus fully resolves *9* (= 4.5%)
