A-CODE-LLM Bench: Agentic Coding Benchmark
We benchmarked the top Large Language Models (LLMs) across 10 software development tasks using an agentic CLI tool. We executed ~3,500 automated validation steps per model across both API and UI layers.
A-CODE-LLM Bench results
Each alias ran 3 times across 10 tasks (30 samples per alias, 380 cells per iteration across 38 aliases). See more details on methodology.
- Mid-tier Sonnet beats the flagship Opus. Both Sonnet versions outscore every Opus, including Opus 4.8 (0.702). Anthropic’s most expensive tier is not its best coder.
- The top is no longer Anthropic-only: Grok 4.5 (0.732) beats every Opus variant. OpenAI’s new flagship GPT 5.6 Sol posts the company’s best score (0.615), still 0.157 below Sonnet 5, and its higher-compute pro variants mostly land below their own bases (Sol Pro 0.543, Terra Pro 0.568; only Luna Pro improves, 0.603 vs 0.579).
- Code-specialists did not win the coding benchmark. GPT 5.3 Codex, OpenAI’s code-tuned variant, scores 0.572, mid-pack and below OpenAI’s own general GPT 5.4 Mini (0.594). Moonshot’s Kimi K2.7 Code is the stronger specialist at 0.611.
- No model is reliable on backend: the ceiling is 0.701 (Sonnet 5), so even the winner fails about a third of the business-logic and contract checks. Grok 4.5 comes closest among the July additions at 0.663. Frontend is near-solved among the leaders (0.79 to 0.96), so backend is the open problem, and it sets the ranking. Claude Haiku 4.5 renders fine (0.731), but a 0.277 backend holds it to 0.413.
- GPT’s weak spot is the frontend. GPT 5.4 and 5.5 match Opus 4.8 on backend (about 0.6) but score 0.53 to 0.55 on frontend; the GPT 5.6 family lifts that to 0.63 to 0.71, still far below the Sonnet line’s 0.91-plus.
Cost & success comparison
- The flagship-priced models are the worst value. Opus 4.7 is the most expensive ($3.08/cell) and scores 0.610, below Sonnet 4.6 at $1.33.
- The top charges a steep premium for little gain: Sonnet 5 scores 0.024 above Sonnet 4.6 for 70% more cost per cell.
- Grok 4.5 is the new best value: 0.732, within 0.040 of the winner, at $0.46 per cell, compared to Sonnet 5’s $2.23. Inside the GPT 5.6 family price buys nothing: $0.18 (Luna) to $2.76 (Sol Pro) per cell for scores between 0.543 and 0.615, with the cheapest alias outscoring the most expensive.
Task completion time & success comparison
- The top score is among the slowest. Sonnet 5 takes about 30 minutes per task, 3x Sonnet 4.6 for 0.024 more; Sonnet 4.6 gives nearly the same score in a third of the time.
- A long run usually signals a stuck model, not a thorough one: the bottom scorers, both Qwen variants, GLM 5.1 base, and Deepseek V4 Pro, each ran over 1,700 seconds from over-iteration for scores below 0.45.
- Grok 4.3 was fast because it quit early: 142 seconds and 18 tool calls for 0.431. Grok 4.5 keeps the speed and drops the quitting: about 9 minutes per task, under a third of Sonnet 5’s time, for 0.732.
Tool calls per task
- Tool-call count measures neither capability nor effort you can compare. Sonnet 5 made the most calls (125) and scored highest; MiniMax M3 made 108 for a mid-pack 0.583; Grok 4.5 reached 0.732 on 40; OpenAI’s low 16 to 36 come from apply_patch bundling a whole file into one call. Sol Pro and Terra Pro call a quarter to a third fewer tools than their bases and score less: more reasoning, less execution. Do not rank agents by tool volume.
- Two paths reach the same score: Sonnet 5 iterates heavily (125 calls), Sonnet 4.6 barely (about 50), 0.024 apart.
LLM performance on a single successful task
No model passed every step of the full benchmark above. To compare cost and speed on equal terms, we ran a simple baseline task that every model can complete: four CRUD endpoints, basic validation, no authentication, and no database.
Cost & lines of code comparison
- Simple tasks cannot rank models, so toy evaluations mislead. On the baseline every model passes, code converges to 40 to 64 lines, and cost drops to cents; differences appear only on long, multi-file work.
- The “fast and light” tier was the most expensive here: Gemini 3.5 Flash base wrote 131 lines for the trivial task, two to three times the field, making it the priciest baseline, against its own positioning.
- Sonnet 5’s heavy iteration is task-driven, not a habit: 9 calls and $0.09 here versus 125 calls on the benchmark.
See more details in the LLM Pricing article.
Completion time & token usage
- Cost predictability splits models in two. Adaptive models spend only when needed (Opus 4.8: 34s baseline, 1,072s benchmark); fixed-pace models run slow and costly even on trivial work (MiniMax M3: 475 vs 1,684s).
- Output length is a fixed model trait, ranging nearly 10x for the same task (787 to 7,508 tokens), feeding directly into cost.
What are agentic LLM systems?
Building software is iterative: write code, run it, read errors, fix them, repeat. Agentic AI systems enable LLMs to follow this same cycle. The model operates inside a development environment where it can write files, execute commands, read outputs, and make changes based on what it sees, continuing until the task is complete.
This matters because real applications aren’t single files. They have backends with routes and database models, frontends with components and API calls, configuration files, dependencies, and tests. Making these work together requires iterative testing and refinement, which is exactly what agentic architecture enables.
How it works
The model sits inside a harness with access to a shell, file system, and execution output. When asked to build an application, it writes files incrementally. After each step, the harness shows the model what happened: did the server start, did tests pass, did the linter flag errors? Based on that feedback, the model decides what to write or fix next.
This differs fundamentally from single-shot generation. In one-shot setups, the model generates an entire codebase blind, with no way to verify if it works. In agentic LLM systems, the model sees the consequences of each action and course-corrects. However, this capability alone isn’t sufficient. The model still needs strong reasoning to implement business logic correctly, which is where performance differences really emerge.
Add as preferred sourceAgentic LLM benchmark methodology
We used Opencode as the agent harness for all models and connected them through OpenRouter, with one exception: Claude Fable 5 ran on the Claude Code CLI on the Claude subscription. Each cell was run 3 times to measure per-cell variance and stabilize the leaderboard. We evaluated their ability to work autonomously on 10 software development tasks (T-1 to T-10), ranging from reservation systems to interactive dashboards. These tasks require agents to manage multi-file projects and deliver functional products. The seven aliases added in July 2026 (Grok 4.5 and the GPT 5.6 family: Sol, Terra, Luna, each in base and pro mode) ran on Opencode 1.15.13 at API-default settings, the same as every model here; OpenAI’s launch numbers for Sol use higher-compute modes.1 Four Sol Pro cells and one Terra Pro cell ended in repeated silent API stream failures rather than model errors and are scored as failures.
Execution and orchestration
Every agent and task begins in a clean environment. The instructions are provided as a TASK.md file, and we use a 20-minute heartbeat watchdog for the launch scripts. During this phase, we record exit codes, execution time, and whether the backend and frontend files were created. We also track real-time token usage across input, output, and cached categories.
Backend validation: We deploy the generated projects in isolated environments to test them against a canonical YAML contract. The validation covers happy path scenarios, error handling (400/403/409), and data consistency.
We test the results in two modes:
Adaptive mode validates functionality even with differing route names, while Strict mode requires exact adherence to the contract.
The backend overall score is calculated per cell as:
backend_overall = has_backend × (0.7 × adaptive_pass_rate + 0.3 × strict_pass_rate)
where has_backend is 1 if the cell produced a backend project, 0 otherwise. Adaptive is weighted higher because it measures behavioral correctness; strict adds a penalty for contract drift (renamed routes, substituted status codes, restructured response fields).
UI and user scenario testing
We use browser automation to simulate real user flows, including preflights, rendering, and authentication. We verify functional steps such as login submission and post-login behavior to ensure the application runs without crashing.
UI scoring splits eight steps into two groups. Infrastructure steps (backend preflight, frontend render, login form visible, login submit, login 2xx, no runtime crash) measure whether the app runs at all. Behavior steps (post-login auth signal, post-login behavior signal) assess whether the app performs its intended function once running.
ui_score = (behavior_passed / (behavior_passed + behavior_failed)) × (infra_passed / infra_total)
Blocked behavior steps are excluded from the behavior denominator, so a cell is not double-penalized when the app fails to load.
Tokens calculation
Token counts are extracted from the LLM API response. We subtract cached input tokens from total input tokens to get the effective input, which reflects only newly processed tokens. Output tokens are never cached, so they remain unchanged.
Final aggregation
The final benchmark score is calculated by combining the results from the previous phases: Final Score = (0.7 × backend_overall) + (0.3 × ui_score) We assign a higher weight to the backend because logic failures at the API level often invalidate any success in the frontend.
Task example
Task 6: Helpdesk ticket system
Task 6 focuses on developing a complex customer support ecosystem. The primary objective is to build a platform that mediates communication between customers and support agents while strictly enforcing business rules and security boundaries. This task evaluates an agent’s ability to handle multi-user state machines, data isolation, and threaded communication within a full-stack environment.
The task required building a helpdesk system featuring:
- Distinct permissions for Customers (issuing/replying) and Agents (management/resolution).
- A rigid status workflow that prevents illegal transitions and enforces role-specific actions.
- Advanced data isolation where unauthorized resource requests return 404 instead of 403 to protect system integrity.
- A chronological reply system for seamless agent-customer interaction.
- A FastAPI backend combined with a responsive Vite-powered frontend (React/Vue/Svelte).
- Reproducible setup via specific shell commands for immediate system activation.
You can view the Task 6 documentation on GitHub.
Cite this benchmark
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{kalelioglu2026,
author = {Kalelioğlu, Berk and Dilmegani, Cem},
title = {{A-CODE-LLM Bench: Agentic Coding Benchmark}},
year = {2026},
month = jul,
howpublished = {\url{https://aimultiple.com/agentic-llm}},
note = {AIMultiple. Retrieved July 10, 2026}
}Results and timestamps of 418 data points. Download the data used in this article as a ZIP file containing 2 CSV files and a README.
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.