Naman Jain

@StringChaos

Research @cursor_ai | CursorBench, LiveCodeBench, DeepSWE, R2E-Gym, GSO, LMArena Coding | Past: @UCBerkeley @MetaAI @MSFTResearch @iitbombay

San Francisco, CA

naman-ntc.github.io

Replies Reposts Media

Pinned
Naman Jain
@StringChaos
Mar 12
New post: how we do evals at @cursor_ai. Takeaways: 1. Online metrics from real Cursor requests provide construct validity 2. CursorBench: a dynamic offline suite distilled from online learnings 3. Multi-axes evals -- correctness, efficiency, agent interaction behavior
Cursor
@cursor_ai
Mar 12
We're sharing a new method for scoring models on agentic coding tasks. Here's how models in Cursor compare on intelligence and efficiency:
5
Naman Jain
@StringChaos
Sep 10
CursorBench got to version 4.0!! With the pace of model improvements, benchmarks need to be living instances constantly updating. We made the benchmark more difficult and longer-horizon in lines with how we use models now!
Lee Robinson
@leerob
Sep 10
We just rolled out CursorBench 4.0! It includes new tasks for how well models follow instructions, work on challenging projects over time, and is more difficult than before (so all models score lower).
1
Naman Jain
@StringChaos
Jul 8
Try our new model built jointly in collaboration with SpaceXAI It provides frontier coding and agentic performance at a much lower total cost. Beyond reduced per token pricing, the model is also more efficient: it can solve complex tasks with fewer steps and tokens!
Artificial Analysis
@ArtificialAnlys
Jul 8
SpaceXAI’s Grok 4.5 scores 54 to place fourth on the Artificial Analysis Intelligence Index following only Fable 5, GPT-5.5, and Opus 4.8. It scores on par with GPT-5.5 in Codex on the Artificial Analysis Coding Agent Index in the Grok Build harness, at much lower cost Grok 4.5
2
Naman Jain
@StringChaos
Jun 25
Coding agents are now smart enough to hack the benchmarks built to measure them. New blog on how frontier models can retrieve known fixes from git history or the public web instead of deriving the solution and why environment design now matters as much as the dataset.
Cursor
@cursor_ai
Jun 25
We're sharing new research on how models hack public benchmarks. The latest models, including Opus 4.8 and Composer 2.5, learn to retrieve solutions from the internet or git history. When we apply a stricter harness, eval scores drop significantly.
2
Naman Jain
@StringChaos
May 18
Check out Composer 2.5, our new model pushing pareto frontier
Cursor
@cursor_ai
May 18
Replying to @cursor_ai
Composer 2.5 is exceptionally intelligent and up to 10x more efficient than similarly capable models.