Kilian Lieret

@KLieret

Meta Superintelligence, prev. Princeton. SWE-bench multilingual/multimodal, SWE-agent, mini-swe-agent, SWE-smith, CodeClash, ProgramBench

New York

github.com/klieret

Replies Reposts Media

Pinned
Kilian Lieret
@KLieret
May 12
The first ProgramBench task was just solved by GPT 5.5 high/xhigh. Interestingly, high/xhigh picked two different languages for the task (C vs Python). GPT 5.5 xhigh was significantly better than Opus 4.7 xhigh in all metrics. 🧵
John Yang and 3 others
35
Kilian Lieret
@KLieret
Sep 1
SWE-bench multimodal is still far from unsaturated!
John Yang
@jyangballin
Sep 1
Replying to @jyangballin
In 2026, debugging GitHub issues communicated w/ visuals (not text) remains partially unsolved. Since Opus 4.7, Anthropic models have steadily climbed in Multimodal performance. Getting to 90+% with this benchmark is just a matter of time.
1
Kilian Lieret
@KLieret
Aug 17
mini-swe-agent with Opus 5 is significantly ahead on TerminalBench 3/FrontierBench. Interestingly it even beats Fable 5 + Claude Code.
3
Kilian Lieret
@KLieret
Aug 14
Had a great time talking at @ScaleAILabs. New levels of model capabilities require new benchmarking paradigms. Also really enjoyed talks by @sirrice @mohit_r9a and @MiguelR33478246! Thanks to all the organizers and to @liuying04
1
Kilian Lieret
@KLieret
Aug 13
Extremely cool project: What's the best SWE-bench score you can achieve by training a model from scratch with limited $ budget. SWE-bench might be near saturated for frontier models, but it still has a lot of life in it for hillclimbing with smaller or from scratch models
Ricardo Olmedo
@rdolmedo_
Aug 13
We turned @karpathy's nanochat into a SWE-bench speedrun! ⚡ $60 of train compute → 5.0% pass@1 ≈ Claude 2 (2023 SOTA) ⚡ $1000 → 11.0% pass@1 > Claude 3 Haiku From randomly initialized model weights! How? 🧵👇