HackerRank-Bench / Technical report
Benchmarking frontier AI models on private, real-world, long-horizon engineering tasks.
HackerRank-Bench is a private, long-horizon benchmark. It tests frontier models on diverse, real-world, complex engineering work across complete software systems. This is the kind of work that public coding benchmarks often miss.
On these tasks, the strongest model completes about half of the verified behavior, and most complete less than 40%. AI can do part of the engineering, but not yet all of it.
Compare each model configuration’s verified score with its cost, duration, or token usage per task.
Score (%)
Cost per task (USD, log scale)
Models do not perform equally well on every kind of task. The top model for migrations is not the top model for frontend work. The charts below show the leading models for each task type.
The five highest-scoring configurations for each task type.
Building a working system from a specification, without an existing implementation to anchor decisions.
Extending an existing codebase while preserving behavior that other parts of the system already depend on.
Reproducing expected behavior across an implementation, framework, or runtime boundary.
Implementing standards-based identity lifecycle and synchronization across tenant boundaries.
Building visual, stateful interactions while preserving rendering and input behavior.
Making correct, safe, and complete changes to how sensitive data is handled end to end.
This section describes how we designed the tasks, ran the models, and scored each run. All tasks, verifiers, scoring weights, and environment settings were fixed before any model was evaluated. Each combination of model and reasoning level is treated as a separate configuration.
Every benchmark task combined three separately versioned components.
A product or engineering brief defined the required behavior, operating constraints, and acceptance boundaries without prescribing one implementation.
A versioned, runnable workspace provided the codebase, services, dependencies, data stores, browser tooling, and a consistent initial state.
A hand-written evaluation package contained hidden fixtures, behavioral checks, criterion weights, and the evidence contract used to produce the score.
Each task asks for work across a whole software system, not a single file or function. The prompts, test data, and verifiers stay private, so the solutions are unlikely to appear in any model's training data. The tasks cover six types of engineering work.
Building a working system from a specification, without an existing implementation to anchor decisions.
Extending an existing codebase while preserving behavior that other parts of the system already depend on.
Reproducing expected behavior across an implementation, framework, or runtime boundary.
Implementing standards-based identity lifecycle and synchronization across tenant boundaries.
Building visual, stateful interactions while preserving rendering and input behavior.
Making correct, safe, and complete changes to how sensitive data is handled end to end.
Example task
Build an asynchronous workflow that removes or anonymizes a customer’s personal data across connected systems while preserving required financial records and data belonging to other customers.
Task contract
Accept a customer data deletion request, expose its progress, clean every active data store, and report completion only when the customer’s personal data is no longer available.
Evaluation
The verifier checks behavior through the public API and running services. Different implementations can pass if they satisfy the same contract.
Tests normal, duplicate, concurrent, invalid, and cross-tenant requests.
Checks each active data store and emitted payload for the customer’s personal data.
Runs delayed work and historical events to confirm erased data does not return.
Runs refunds and isolation checks to confirm required records and unrelated data still work.
Every run starts from the same saved workspace, with identical code, services, data, and tools. All models operate through the same agent harness, OpenHands. The verifier is never visible to the model. It is added only after the model finishes, in a fresh container. Each configuration was run three times on every task, because results vary from run to run. Runs that failed because of the model provider or the environment, such as API errors or container crashes, were discarded and repeated. The leaderboard reports the average score across all runs, with a line marking the lowest and highest of the three attempts.
Each verifier evaluates what the software does, not how the code is written. Two kinds of verifier are used, depending on how precisely the task can be specified in advance.
Fixed contract
Used when the task specifies the exact API and expected behavior.
Adaptive contract
Used for open-ended tasks, where two valid solutions may expose different APIs or screens for the same feature.
Verifiers are written by hand by expert engineers and validated before use. A reference solution must receive the full score. We then introduce defects into that solution, one at a time, and confirm that the verifier catches each one. Finally, we run the verifier against multiple independent implementations to confirm that it rewards the same behavior regardless of how the code is structured.
Each task is divided into separate checks, and each check is assigned a weight. Behavior that is more important or harder to get right receives a larger weight. A run’s score is the weighted share of checks it completes.
Task score for one run
Scorer,t=100×NtΣi=1wt,ipr,t,iNtΣi=1wt,iScores are not binary. A partly working system receives partial credit rather than a zero. Credit is awarded only for behavior the verifier directly observes.
The highest score comes at a much higher cost, while GPT-5.6 Luna Medium delivers the strongest score per dollar.
GPT-5.6 Luna Medium averages 18.19% at just $0.18 per run, the best score for every dollar spent. Claude Opus 5.5 High has the highest score at 51.23%. Claude Opus 5 High costs the most at $47.33 per run.
Inspection accounted for 80.83% of recorded Git requests. Models differed in how often they requested staging and commits, while undo and stash operations made up just 0.49% overall.
git status, git diff, git log, git show, git ls-files; configuration reads and other read-only queriesgit init; configuration writes with git configgit add, git rm, git mvgit commitgit restore, git reset, git revert, git checkout for files; stash operations| Model | Inspect | Setup | Stage | Commit | Undo/stash | Other |
|---|---|---|---|---|---|---|
| Claude Opus 5 | 67.44% | 15.35% | 7.91% | 7.91% | 0.00% | 1.39% |
| Claude Opus 5.5 | 77.69% | 12.05% | 5.13% | 5.13% | 0.00% | 0.00% |
| Claude Sonnet 5 | 91.21% | 3.29% | 2.20% | 2.20% | 0.00% | 1.10% |
| DeepSeek V4 Pro | 33.33% | 26.09% | 27.54% | 8.69% | 2.90% | 1.45% |
| Gemini 3.7 Flash | 94.83% | 5.17% | 0.00% | 0.00% | 0.00% | 0.00% |
| GLM 5.2 | 78.79% | 10.61% | 6.06% | 3.03% | 1.51% | 0.00% |
| GPT-6 Astra | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| Grok 4.6 | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| Kimi K3 | 81.96% | 6.56% | 3.28% | 3.28% | 3.28% | 1.64% |
| MiniMax M3 | 100.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| Qwen 3.8 | 88.14% | 5.08% | 3.39% | 3.39% | 0.00% | 0.00% |
The runs show two contrasting rhythms: some repeatedly inspected files, while others moved through short edit-and-check cycles. The balance between those patterns differed across model groups.
Tool calls show how a model works through a frontend task. Each square is 1% of the 2,539 back-to-back tool-call pairs.
read_file → read_file43.4% 1,102edit_file → edit_file24.3% 618edit_file → run_terminal_cmd7.4% 189read_file → edit_file5.5% 140edit_file → build_test3.5% 89build_test → edit_file2.1% 54list_dir → read_file1.9% 48Each row shows a run with the highest score in one model family. Colors identify transition types. Segment widths show each type’s share among the five patterns in that run.
| OpenAIGPT-5.6 Sol (High) | |
|---|---|
| AnthropicClaude Opus 5 (High) | |
| MoonshotKimi K3 (High) | |
| GoogleGemini 3.7 Flash (High) | |
| xAIGrok 4.6 (Medium) | |
| Alibaba / QwenQwen 3.8 (Medium) | |
| MiniMaxMiniMax M3 (High) | |
| Z.ai / GLMGLM-5.2 (Medium) | |
| DeepSeekDeepSeek V4 Pro (Medium) |
In both the Claude and GPT families, the highest-scoring model took fewer steps, created fewer files, and made fewer tool calls.
Opus 5.5 leads this group on mean score, with fewer steps and tool calls than Opus 5 or Sonnet. Its lower read-to-write ratio reflects more implementation work, not skipped exploration: the saved solution includes a layered service and separate role-based screens, and its trajectory includes setup checks and repeated build and lifecycle checks.
| Model | Files | Steps | Tool Calls | Cost | Avg. score |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 57 | 200 | 182 | $18.86 | 51.23% |
| Claude Opus 5 | 136 | 424 | 388 | $47.33 | 42.58% |
| Claude Sonnet 5 | 381 | 397 | 354 | $20.12 | 28.03% |
Astra leads this group on mean score, with fewer steps and tool calls than Sol. Its lower read-to-write ratio reflects more implementation work, not skipped exploration: the solution includes transaction-backed state, route planning, and a role-aware workspace, while its trajectory shows setup checks, builds, tests, and a browser check.
| Model | Files | Steps | Tool Calls | Cost | Avg. score |
|---|---|---|---|---|---|
| GPT-6 Astra | 50 | 96 | 81 | $12.36 | 43.51% |
| GPT-5.6 Sol | 278 | 173 | 146 | $10.76 | 38.43% |
| GPT-5.6 Luna | 367 | 191 | 97 | $0.18 | 26.06% |
Luna’s $0.18 estimated cost reflects high cache reuse. Its listed input and output rates ($0.20 and $1.20 per million tokens) are the lowest among the frontier models compared here.
Scores varied across runs, and some traces spent substantial time repeating checks or revisiting the same work without clear progress.
| Model and reasoning | Problems in the run |
|---|---|
| Qwen 3.8high reasoning |
|
| DeepSeek V4 Promedium reasoning |
|
| Kimi K3medium reasoning |
|
| GLM 5.2high reasoning |
|
| MiniMax M3high reasoning |
|
Given a list of requirements and a prepared environment, agents had to build the entire application from scratch. Some inspected the environment and planned the structure before writing code; others began implementing sooner. Some checked complete workflows as they built, while others concentrated on local edits, repeated builds, and restarts.
| Model, score, and duration | General approach |
|---|---|
| AnthropicClaude Opus 5.5 41.74% score, 95.6 min | Inspected project context before planning, then implemented the system in stages and checked progress as the work continued. |
| OpenAIGPT-6 Astra 30.42% score, 55.1 min | Moved from initial inspection into a structured implementation, with checks across the system as the work progressed. |
| Z.ai / GLMGLM-5.2 20.06% score, 12.8 min | Moved quickly from project inspection into implementation and validation, completing its run in 12.8 minutes. |
| Alibaba / QwenQwen 3.8 25.15% score, 74.5 min | Spent more time exploring project context before implementation and completed fewer end-to-end checks during the run. |
| MoonshotKimi K3 21.22% score, 51.6 min | Interleaved implementation with repeated checks and restarts, adding time without consistent progress. |
| MiniMaxMiniMax M3 20.06% score, 46.1 min | Worked through several implementation and build iterations before moving on to broader system checks. |
| GoogleGemini 3.7 Flash 20.19% score, 11.8 min | Completed its run in 11.8 minutes after a shorter implementation cycle, with fewer checks across the full workflow. |
| xAIGrok 4.5 17.03% score, 34.0 min | Adjusted to initial project constraints, then continued with implementation and system checks. |
| DeepSeekDeepSeek V4 Pro 12.46% score, 29.4 min | Used a plan-first approach and worked through several implementation and validation cycles. |
High reasoning helped on brownfield tasks, such as code migration and SCIM integration, where agents worked with an existing codebase. On greenfield tasks, where the application had to be built from scratch, increasing reasoning effort produced only a small score gain. For the greenfield application, the relative increase in wall-clock time was roughly twice that of SCIM integration (16.1% versus 8.4%).
| Task | Median score change | Median wall-clock change |
|---|---|---|
| SCIM integration | +7.86 pp | +8.4% |
| Building a full-stack application from scratch | +1.99 pp | +16.1% |
Across system integration and interactive application development, results show room to improve in workflow completeness, state management, and interaction accuracy.
| Task type | Behavior area | Observed result | What it suggests |
|---|---|---|---|
| System integration | Workflow completeness | 47.2% of points earned | Completing an entire workflow was the largest observed gap. |
| System integration | Input handling and consistency | 54.6% of points earned | Handling invalid inputs and related updates remained inconsistent. |
| System integration | State management | 58.0% of points earned | Behavior across state changes was not consistently implemented. |
| Interactive application development | Interaction accuracy | 38.8% of points earned | Detailed interactions were less reliable than basic actions. |
| Interactive application development | State and content handling | 40.4% of points earned | Managing prior state and moving content remained inconsistent. |
Total cost reflects token volume, model rates, and cache reuse, so lower token rates alone do not guarantee a lower bill.
In one specific comparison, we noticed that Qwen 3.8’s modular backend and GPT-5.6 Sol’s single-file backend earned the same 12.45% score. Qwen spread the backend across 18 production Go files, while Sol kept it in 1 file and still handled edits well enough to match the behavioral result. We report this outcome without endorsing the single-file approach.
| Model | Backend structure | File-editor mutations | Verified score | Detailed checks passed |
|---|---|---|---|---|
| Qwen 3.8 | Separate API, auth, domain, storage, and planning packages (18 production Go files) | 168 | 12.45% | 8/125 |
| GPT-5.6 Sol | Service behavior concentrated in one server file (1 production Go file) | 27 | 12.45% | 8/125 |
Separating responsibilities into modules can make code easier to navigate, debug, and maintain. Qwen 3.8 used separate API, authentication, domain, storage, and planning packages, while GPT-5.6 Sol concentrated service behavior in one server file. That structural difference did not change the measured outcome: both passed the same eight detailed checks. A behavioral score can therefore leave maintainability concerns and code-quality standards unexamined; matching scores do not establish equally maintainable code.
The table shows the percentage of each model’s implementations that passed the REST or WebSocket checks. Most models reached 100% on REST checks, while pass rates for WebSocket checks ranged from 0% to 75%. Every model had a lower pass rate on WebSocket checks, suggesting a shared weakness in implementing stateful, live connections.
| Model | REST checks | WebSocket checks |
|---|---|---|
| Claude Opus 5 | 100.0% (6/6) | 33.3% (2/6) |
| Claude Opus 5.5 | 100.0% (6/6) | 40.0% (2/6) |
| Claude Sonnet 5 | 100.0% (6/6) | 50.0% (3/6) |
| DeepSeek V4 Pro | 100.0% (6/6) | 75.0% (5/6) |
| GLM 5.2 | 100.0% (6/6) | 66.7% (4/6) |
| GPT-5.6 Luna | 100.0% (6/6) | 50.0% (3/6) |
| GPT-5.6 Sol | 100.0% (6/6) | 16.7% (1/6) |
| GPT-5.6 Terra | 100.0% (6/6) | 60.0% (4/6) |
| GPT-6 Astra | 100.0% (6/6) | 0.0% (0/6) |
| Gemini 3.7 Flash | 100.0% (6/6) | 60.0% (4/6) |
| Grok 4.5 | 100.0% (6/6) | 0.0% (0/6) |
| Grok 4.6 | 100.0% (6/6) | 40.0% (2/6) |
| Kimi K3 | 100.0% (6/6) | 0.0% (0/6) |
| MiniMax M3 | 80.0% (5/6) | 40.0% (2/6) |
| Qwen 3.8 | 80.0% (5/6) | 20.0% (1/6) |
The results should be interpreted within the scope of this benchmark and its execution protocol.
470M graded submissions, across 2000+ questions, spanning 30+ programming languages.
This is community driven human data that developers use to prepare for interviews.
Every month we host Orchestrate, a community hackathon where 30k+ participants take on a complex long horizon SWE challenge.
Gives us trajectories on the performance of the latest frontier models on complex software engineering.
Executable environments for repository-scale, long-horizon agent training and evaluation.
Built on HackerRank’s experience creating practical, verifiable engineering assessments.
HackerRank can run benchmarks on long horizon problem sets that have never touched the public internet, each paired with real human expert performance trajectories wherever required.
