Kilian Lieret

@KLieret

Meta Superintelligence, prev. Princeton. SWE-bench multilingual/multimodal, (mini-)SWE-agent, SWE-smith, CodeClash, ProgramBench, SWE-sweep

New York

github.com/klieret

Joined May 2021

295 Following

3,008 Followers

Replies Reposts Media

Pinned
Kilian Lieret
@KLieret
Oct 1
Can LMs discover & fix bugs if you don't tell them what went wrong? To proactively maintain a repo, agents need to find problems before users do. SWE-sweep benchmarks this on 100 repos, 22 languages, 4k real bugs (numpy, php interpreter, lean kernel). Top models get <5%.
John Yang and 9 others
75
Kilian Lieret
@KLieret
Oct 28
SWE-sweep is the least saturated benchmark on the model card. New SoTA is 9.6%! Congrats to the Gemini team
Sundar Pichai
@sundarpichai
Oct 28
I did say we'd be releasing our models periodically...introducing Gemini 4 Carbon, our next frontier model. Following Argon's release just two weeks ago, Carbon shows the pace at which we’re pushing the frontier, it ranks #1 on the recently updated v5.0 of @ArtificialAnlys
Kilian Lieret
@KLieret
Oct 28
Congratulations to the Gemini team on reaching SoTA on SWE-sweep, scoring 9.6% (narrow leaderboard over Astra 6)! Hoping to update leaderboard with details soon!
Logan Kilpatrick
@OfficialLoganK
Oct 28
Introducing Gemini 4 Carbon, it's a big jump over Argon which we released just 2 weeks ago. Gemini 4 Carbon is SOTA on cyber, long horizon tasks, ranks #1 on Artificial Analysis V5, significantly improves on usability in domains like coding, and is more aligned.
1
Kilian Lieret
@KLieret
Oct 28
Very interesting study about proactive bug finding agents. Recall and precision are with respect to a holdout set of bugs that humans found, findings is the total number of bugs found
Mark Müller
@mnmueller
Oct 28
New paper: Frontier Coding Agents Find Different Bugs Than Maintainers Fix In proactive bug finding, agents report fewer than 20% of the bugs humans fixed. But they report a lot of other bugs, most of which are valid. This points towards an important distributional mismatch.
2
Kilian Lieret
@KLieret
Oct 27
Progress on ProgramBench is crazy! When it was launched, every model had a flat 0% score and we were worried enough to introduce weaker metrics ("almost resolved" etc.), but things have certainly changed!
4

Kilian Lieret

Kilian Lieret’s posts