Pinned
The first ProgramBench task was just solved by GPT 5.5 high/xhigh. Interestingly, high/xhigh picked two different languages for the task (C vs Python). GPT 5.5 xhigh was significantly better than Opus 4.7 xhigh in all metrics. 🧵
Kilian Lieret
@KLieret
Meta Superintelligence, prev. Princeton. SWE-bench multilingual/multimodal, SWE-agent, mini-swe-agent, SWE-smith, CodeClash, ProgramBench
- Thanks to @agihouse_org for hosting me for a research spotlight! In the talk I look at agentic coding benchmarks, starting with SWE-bench and then covering CodeClash and ProgramBench (both done together with the amazing @jyangballin)Replying to @agihouse_org and @KLieretFull talk: youtu.be/wOtS8W-_QKs?si…
- mini-swe-agent v2.4.6 fixes a bug that could result in unlimited spending if 1) your LM is caught in a loop of malformated outputs and 2) you disabled stopping after n malformats
- GLM _almost_ got its first instance solved. Are there any other models other than Opus/GPT that managed to solve a full instance? Submissions to the leaderboard are open!Replying to @parth007_96On the cmatrix task, the first ever fully solved ProgramBench task (by GPT 5.5), it came up just a *single* test shy of solving it (505/506). cmatrix has a lock mode (the -L flag): run it and it "locks" your terminal, like an old-school screensaver, and prints the words
- Back at evaluating more models on ProgramBench! Very important to look at the number instances that are almost/fully resolved instead of simply averaging test scores. Even a few failed tests can indicate large shortcomings, so reporting >80% average test pass rates is misleading.Official ProgramBench leaderboard update: Our first open weight model we evaluated, GLM 5.2, scores an impressive 8.5% almost resolved, achieving 3rd place overall.
