CodeRabbit Blog

Opus 5 for code review: Cleaner actionable comments, noisier overall

Hendrik Krack — Fri, 24 Jul 2026 00:00:00 GMT

Anthropic just released Claude Opus 5, the next major version of the Opus family many of us have grown to love (at least until Fable stole the spotlight). It's a good moment to step back and look at how Opus has evolved across our testing at CodeRabbit. ![Step chart tracing the Opus family from Opus 4 to Opus 5: each release added actionability, system context, coverage, then coverage parity with the baseline; Opus 5 turns toward precision at a coverage cost.](/content/assets/opus-5-family-evolution.png) We've put every Opus release on the bench since version 4, so we have a pretty good picture of how it's evolved. Opus 5 is the first release where the story isn't simply "it finds more bugs." (Want the full history? Start with [4.1](https://www.coderabbit.ai/blog/claude-sonnet-45-better-performance-but-a-paradox), then [4.5](https://www.coderabbit.ai/blog/opus-45-for-code-related-tasks-performs-like-the-systems-architect), [4.7](https://www.coderabbit.ai/blog/claude-opus-4-7-for-ai-code-review), [4.8](https://www.coderabbit.ai/blog/opus-4-8-release), and [Fable 5](https://www.coderabbit.ai/blog/fable-5-model-review).) ## **What's new in Opus 5** The release is less of a raw-capability bump than it is a shift in how the model is controlled and where it spends its effort. Five changes matter for review and coding work: - **Effort becomes the primary control.** Thinking is on by default and can only be disabled at high or lower effort. Task and effort budgets can now be changed per-turn, mid-conversation without invalidating the cache. Our data backs the framing: Effort was a routing decision — x-high bought a precision gain at a coverage cost, and nothing improved uniformly. - **Long context, claimed stable.** The context window moves to 1M tokens as both default and maximum, with performance claimed to hold throughout. This directly addresses the [200k-token degradation we reported on Opus 4.8](https://www.coderabbit.ai/blog/opus-4-8-release). - **Long-horizon agentic work is the headline strength.** Guidance recommends a complete task specification up front, longer autonomous sessions, and multi-agent writer-verifier patterns. - **Behavioral shifts that change prompts.** The model follows review instructions literally (conservative language suppresses recall), produces longer default output that responds to prompting rather than the effort setting, verifies its own work without being asked, and can expand task scope. Three of those four show up in our data, unpacked further down. - **Elevated cybersecurity safeguards.** As we saw with Fable 5, benign security work can occasionally trigger them. We flag this below in the security-category results, though it happened less frequently with Opus 5. ## **What we tested** The comparison covers about 100 common error patterns (EPs), drawn from verified issues in real open-source pull requests. We ran each configuration three times and compared the averages with three runs of our current production model mix. The comments in this report are post-pipeline outputs, meaning they have already passed through verification, deduplication, and assertive filtering. Four measurements matter here. Two of them ask "did the model catch the bug?" (counted per known issue); two ask "was the comment worth reading?" (counted per comment): - **Actionable pass rate:** share of EPs caught by a confident, actionable comment. - **Full-stream pass rate:** share of EPs caught by any comment class, including outside-diff and assertive/nitpick (low confidence). - **Actionable precision:** share of actionable comments that hit their target issue. - **Full-stream precision:** the same, but across every comment the model produced. ## **TL;DR** Opus 5 is not the best general-purpose code reviewer we tested. Its x-high configuration produced a more precise actionable-comment stream than our production baseline, 39.3% versus 35.2%, but caught fewer of the benchmark's known issues, 55.2% versus 61.1%. It also generated roughly four times as many nitpicks, and its precision fell below baseline when we counted the full post-pipeline stream, 28.6% versus 32.8%. ![Four bar charts comparing the production baseline with Opus 5 x-high: known issues caught 61.1% versus 55.2%, actionable precision 35.2% versus 39.3%, nitpicks 23 versus 92, and full-stream precision 32.8% versus 28.6%.](/content/assets/opus-5-vs-baseline-tradeoff.png) Our read is deliberately narrow: Opus 5 may become a useful precision-oriented lane inside a routed ensemble, but these results do not support using it as the only reviewer or as the primary safety net on high-risk changes. We ran three CodeRabbit review configurations: - Junior-reviewer profile at the default effort level (medium) - Senior-reviewer profile at high effort - Senior-reviewer profile at x-high effort ![Results table of three-run averages across the same evaluation patterns, comparing the production baseline with the three Opus 5 configurations on pass rates, comment precision, and comment volume.](/content/assets/opus-5-results-table.png) ## **What this means for developers** Opus 5 x-high is not a straightforward upgrade to the production reviewer. It produced the cleanest actionable subset, 39.3% precision versus 35.2%, but caught fewer known issues and generated four times as many nitpicks. For developers, that means just a slightly better chance that an actionable comment is useful, but a greater risk that important issues are missed and more low-value feedback needs triage. The practical fit is as a second, precision-oriented reviewer. More reasoning did not consistently produce a better review. The junior/default configuration found the most issues when every comment class was counted, but its full-stream precision fell to 26.4% and it produced 110 nitpicks. X-high was more selective among actionable comments, yet remained noisy overall. Teams should therefore treat effort as a choice between failure modes: Decide whether the workflow needs broader exploration or a cleaner actionable subset, then measure the filtering burden each choice creates. ![Table of category-level signals for Opus 5: promising on configuration errors and code quality, mixed on null references, weaker on logic errors, API misuse, and data validation, and clearly weaker on race conditions.](/content/assets/opus-5-category-signals.png) Opus 5 is strongest on configuration errors and code quality. It is good at spotting integration details and maintainability problems, although that broader review style also produces more noise. Its weaker areas are logic errors, race conditions, and API misuse. That makes Opus 5 useful as an additional perspective, but not the only reviewer for correctness-heavy changes, especially around concurrency or API behavior. ## **How Opus 5 fits into the recent model landscape** Historical CodeRabbit results provide context, not a leaderboard. The reports used different EP counts, model mixes, judges, prompts, and pipeline versions, so small numeric differences should not be treated as head-to-head wins. ![Scatter plot of coverage versus actionable precision placing GPT-5.6 Sol at the recall-oriented end, Opus 5 x-high, Sonnet 5, and GPT-5.6 Terra on the precision-oriented side, and the production baseline and Opus 4.8 in between.](/content/assets/opus-5-model-landscape.png) The frontier models have split into two camps: recall-oriented reviewers that catch more and lean on the product to clean up after them, and precision-oriented reviewers that say less and are right more often. [GPT-5.6 Sol](https://www.coderabbit.ai/blog/gpt-5-6-sol-and-terra-benchmark) anchors one end of the spectrum with 69.7% of known issues caught, but only 31.6% of its comments were worth keeping. Opus 5 x-high anchors the other end of the spectrum with the cleanest actionable stream on the chart at the lowest coverage of any Opus lane we have tested. Its nearest neighbor is [Sonnet 5](https://www.coderabbit.ai/blog/claude-sonnet-5-review), which reached similar precision earlier. The difference is that Opus 5 holds slightly more coverage, while producing a noisier full stream. Opus 4.8, sitting next to the baseline, remains the family's balanced option. ## **Where Opus 5 shines: Code generation** Opus 5 makes a stronger case as a builder than as a reviewer. One of our engineers described it as "less anxious" than earlier Opus models. It takes time to clarify the goal, offers several credible approaches, and handles open-ended projects without rushing into the first solution. It also documents its work thoroughly, which improves explainability, but can consume a lot of tokens. On larger agentic tasks, Opus 5 coordinated hundreds of background agents and showed much better design judgment than Opus 4.8. It was still slower and less efficient than Fable 5, occasionally hit limits, and remained overly cautious around security-related tasks. Our early read: It is a clear step up from Opus 4.8, especially for ambiguous, design-heavy projects, but it is not a Fable contender. ## **Token consumption per review call** ![Bar chart of tokens per review call: Opus 5 averages roughly 60,500 input and 9,500 output tokens, versus roughly 40,500 input and 5,800 output tokens for the production baseline (the GPT-5.6 lanes) in the same runs.](/content/assets/opus-5-token-consumption.png) For every review call, Opus 5 reads about 50% more and writes about 65% more than the frontier models in our baseline. It averaged ≈60.5k input tokens per call (55k to 64k depending on configuration) and ≈9.5k output tokens (8.1k to 10.8k), against ≈40.5k in and ≈5.8k out for the GPT-5.6 lanes doing the same job in the same runs. In practice, every Opus 5 review call carries a bigger context in and returns a longer answer back, which is where the comment volume, and part of the cost premium, comes from. X-high was the heaviest writer, producing 10.8k output tokens per call. More effort did not automatically mean more output: High wrote less than junior/default, while x-high increased both thinking and output. ## **How to evaluate Opus 5 yourself** Don't judge it on one headline score. Opus 5 can produce cleaner actionable comments while missing more known issues and creating more noise overall. Your evaluation needs to capture both what it finds and what developers must sift through. - Test every relevant effort level, including low and medium. More reasoning changed the trade-off in our runs; it did not consistently improve the review. - Compare restrictive prompts with broader reporting followed by filtering. Instructions such as "only report high-severity issues" may suppress useful findings. - Tune conciseness and severity before rollout, and before developers learn to skim the review. - Remove redundant verification instructions and watch their effect on runtime, tokens, and cost. - Break down misses by category and severity. If missing an issue is the expensive failure, pair Opus 5 with a recall-oriented reviewer. ## **Our verdict: Give it a specific role** Opus 5 is easier to recommend as a builder than as a standalone reviewer. For ambiguous, design-heavy coding tasks, it is a meaningful step up from Opus 4.8. It's more deliberate, better at exploring options, and more capable of coordinating complex work. Fable 5 still looks stronger and more efficient for long-running orchestration. For code review, Opus 5 x-high is a specialist. It produces a cleaner actionable subset, but catches fewer known issues and adds a substantial nitpick tail. Use it where precision matters, pair it with recall-oriented coverage, and filter its output before it reaches developers.

Code is no longer the bottleneck. Understanding is.

Harjot Gill — Tue, 21 Jul 2026 00:00:00 GMT

Coding agents have changed what is scarce in software. A plausible change takes less time to produce. A team still needs to decide whether it belongs, how it reshapes the system, and what should happen next. Passing tests do not answer those questions. This is the explainability gap. Models can take on more context and work in parallel. Human comprehension does not scale the same way. The bottleneck moves from producing code to understanding it well enough to direct it. When a team can no longer explain a change, it stops shaping the system and starts accepting output. Most review interfaces start with files, lines, and comments. That evidence is essential. It is also a poor place to begin when one system change crosses routes, providers, templates, tests, and configuration. Review needs a higher level of abstraction, but not a summary that asks for blind trust. It needs a path from intent to system behavior to code, with every claim traceable to the diff.

## Understanding keeps judgment active A reviewer is not there to count changed files. The job is to answer a different set of questions: - What changed at the system level? - Which parts of the system move together? - Where are the boundaries and assumptions? - What deserves human attention? - Should this design become part of the system? Tests, static analysis, and agents can verify more implementation detail. People still decide whether a boundary is sound, whether a tradeoff is worth it, and whether the change points the system in a useful direction. Without a working model of the change, a reviewer can only approve or reject. With one, they can improve the design, challenge an assumption, or connect the change to the next decision. Reviewers should not each have to reconstruct the change from a file tree. That work is slow, and separate reconstructions make discussion harder. Review should give the team a shared model to test. ## A diff is evidence, not a route A repository path tells a reviewer where code lives. It does not explain why several files changed together or which file should be read first. Repositories group code by implementation. Reviewers reason about behavior and decisions. One behavior can cross a route, provider, configuration file, template, and test. The diff remains the exact evidence. It needs a useful reading order. ## One change, two reading orders CodeRabbit recently introduced Change Stack, a review view that groups related edits into named paths based on system behavior, then links each path back to the exact files and lines. The [TanStack/cli pull request #490](https://github.com/TanStack/cli/pull/490) shows it on a real change spanning 45 files across authentication, environment handling, templates, package configuration, and tests.

Same change, higher altitude One diff, two reading orders: file structure and system behavior.

GitHub presents the pull request as 45 changed files. Change Stack organizes the same diff into six review paths based on related system behavior. Neither view changes the code. They change where the reviewer starts.

File organization GitHub preserves the exact file and line history. That is the evidence a reviewer must be able to inspect.

Change organization Change Stack proposes a reading order based on related behavior. Each path still opens the code beneath it.

The two views answer different questions. GitHub shows **where the code changed**. Change Stack proposes **which changes should be judged together**. Those six paths are not a verdict. They are a claim about which code ranges belong in one thought, and reviewers can challenge that claim. ## The useful unit is behavior One Change Stack path joins ten files into a single sign-in flow. No one file contains the behavior the reviewer needs to judge. It exists between the route, provider, callback, configuration, and tests.

System behavior Ten files form one request path. The diagram shows how control moves before the reviewer opens the implementation.

The diagram gives the reviewer a model to test against the code. The first question becomes not “what is this flow?” but “is it correct, complete, and safe?” The author, reviewer, and maintainer can challenge the same proposed flow instead of carrying three private interpretations of the diff. ## A map must lead back to code Abstraction is cheap when it ends at a polished summary. Traceability is harder. In this review, the reviewer can move from the pull request to a change layer, from the layer to a system flow, and from the flow to semantic entities, files, and lines. They can judge the behavior at a high level, then test each claim against the implementation.

Zoom out to recover intent and behavior. Zoom in to verify the map against code. The abstraction is useful only when both directions remain open.

Each level has one job. Intent explains why the change exists. System behavior shows how parts interact. A layer sets a review boundary. A semantic entity names the function, route, or type involved. Files and lines provide the evidence. Remove the upper levels and the reviewer must reconstruct the system from coordinates. Remove the lower levels and the reviewer is asked to trust a story they cannot verify. ## The review interface becomes a stack As coding agents produce more, larger diffs and longer summaries will not make the work governable. Both leave the reviewer to reconstruct the system. A useful review model can be inspected, corrected, and discussed. The review interface needs a stack, not a prettier summary. Each level answers a different question, and every level connects back to code. No representation is sufficient alone. The value comes from moving between intent, behavior, and exact evidence without breaking the chain. The goal is not to read less code. It is to spend code-reading time testing a coherent model instead of constructing one from scratch. Machines can organize the change and expose the paths. Humans decide whether the model is accurate, the boundary is sound, and the direction is worth taking. The diff stays the ground truth. The stack makes it usable for judgment. [Inspect the merged pull request on GitHub](https://github.com/TanStack/cli/pull/490) · [Open the same pull request in Change Stack](https://app.coderabbit.ai/change-stack/TanStack/cli/pull/490)

Close the loop after every merge: the agent that reviewed your PR can now follow through

Sahana Vijaya Prasad — Thu, 16 Jul 2026 00:00:00 GMT

After a pull request merges, a team may still need to write the changelog entry, update the right documentation section, or create a follow-up ticket for work that did not belong in the original change. By then, the context behind the decision can be harder to reconstruct. Some follow-through is mechanical: a linked ticket can close, a version can bump, or a notification can send. Other work depends on understanding what the pull request changed and whether that change is worth documenting, communicating, or tracking. [Post-Merge Actions](https://docs.coderabbit.ai/pr-reviews/post-merge-actions) give the CodeRabbit agent that context. They run after a pull request is merged to the default branch and can carry the change through the work that follows. ## What happens after merge Post-Merge Actions are custom automations that CodeRabbit evaluates during the pull-request review. Before merge, every enabled action appears in the CodeRabbit walkthrough as a checkbox. Applicable actions are checked; actions that do not apply, or whose applicability is unclear, remain unchecked. Authors and reviewers can also opt out of an action before merging. After the pull request merges, CodeRabbit runs the checked actions and posts a single result comment on that merged pull request. The comment records each action's outcome and includes details about any follow-up output. ![Post-Merge Actions workflow: review a pull request, see each enabled action as a checkbox, merge, then receive a result comment with each action's outcome.](/content/assets/close-the-loop-after-every-merge-workflow.png) Post-Merge Actions are the merge-time counterpart to [Pre-Merge Checks](https://www.coderabbit.ai/blog/pre-merge-checks-built-in-and-custom-pr-enforced). Pre-Merge Checks evaluate requirements before a change lands; Post-Merge Actions use the same pull-request context for follow-through after it merges. Admins can configure actions at the organization or repository level. ## What an action can do Each action uses a prompt to decide whether it applies and what it should do. Its output can take one of three forms: - **Code changes:** CodeRabbit edits or creates files and opens a follow-up pull request against the default branch. For example, it can update an entry in the Unreleased section of `CHANGELOG.md` after a user-facing change. - **A text report:** CodeRabbit writes a free-form summary directly in the result comment on the merged pull request. - **An integration action:** CodeRabbit uses a connected MCP tool to create a ticket, open an issue, or post a notification in the system where that follow-up belongs. Follow-up pull requests created by code-change actions do not start another round of Post-Merge Actions when they merge. That keeps the workflow reviewable and prevents accidental recursion. ## What it feels like in a pull request The workflow starts where engineering work already happens. CodeRabbit reviews the pull request, then surfaces the actions that are relevant to that specific change in the walkthrough. A documentation action might be checked for a new public feature, but remain unchecked for an internal refactor. A release notification might be checked only when files in a product area changed. That decision happens before merge, when the author and reviewers can still see the complete context. They can keep an action selected, clear it, or skip a specific action with a chat command. The result is a deliberate handoff from review to follow-through, rather than a background job that runs on every merged pull request. ## Set up an action You can configure Post-Merge Actions in CodeRabbit settings at the organization or repository level, or version them in `.coderabbit.yaml`. Each action has a name, an enabled state, and a natural-language prompt. The prompt is the operating policy. It should say what to inspect, what to change or create, and when to do nothing. Specific instructions make actions more useful and less noisy. ```yaml reviews: post_merge_actions: - name: "Update changelog" enabled: true prompt: "If this PR contains user-facing changes, append a concise entry to CHANGELOG.md under the Unreleased section. Skip internal refactors, tests, and CI-only changes." ``` For a ticketing action, name the conditions that count as unfinished work. For a documentation action, name the exact files to inspect and the sections that may need updating. Add a clear no-op condition so the agent does not force a follow-up when the change does not need one. The full configuration reference covers YAML and dashboard setup, action limits, and the commands reviewers can use to skip actions for a pull request. ## Use connected tools when the work belongs elsewhere Post-Merge Actions can use connected MCP servers when the result belongs beyond the repository. For example, an action can create a Linear or Jira ticket for a documented follow-up, open a GitHub issue, or post a release notification to Slack. The relevant server must be configured and its required tools enabled in CodeRabbit first. [MCP integrations](https://docs.coderabbit.ai/integrations/mcp-servers) also let you give CodeRabbit guidance on what each connected system contains and how the agent should use it. That extra context matters when a project tracker or knowledge base contains much more than the information an action needs. ## Start with the work that needs PR context Post-Merge Actions are most useful when the right next step depends on the change itself: - Does this user-facing change deserve a changelog entry? - Which documentation section changed with this API or workflow? - Did the pull request leave a well-described follow-up that should become a ticket? For a mechanical trigger that does not need pull-request context, consider using your existing CI or webhook automation. Use Post-Merge Actions for the follow-through that depends on the review context the team would otherwise lose after merge. Post-Merge Actions are included in [CodeRabbit Pro+ and Enterprise plans](https://docs.coderabbit.ai/management/plans). **[Read the full Post-Merge Actions documentation →](https://docs.coderabbit.ai/pr-reviews/post-merge-actions)**

GPT-5.6 Sol and Terra: Where they fit for coding agents and code review

Juan Pablo Flores — Thu, 09 Jul 2026 00:00:00 GMT

OpenAI has released [GPT-5.6](https://openai.com/index/gpt-5-6/), and the practical question for engineering teams is where Sol and Terra fit in the coding stack. This review looks at the coding experience, long-running agent work, and CodeRabbit review benchmarks to separate where each model is useful from where the older routing playbook still holds. If you use [GPT-5.5](https://www.coderabbit.ai/blog/gpt-5-5-benchmark-results) or a previous OpenAI model for coding agents, start testing Sol. It follows through better. It works through messy repo tasks, takes long checklists seriously, and finds more review issues when we run it in our harness. Terra is the cheaper lane to test for scoped work. Its long coding run is a reminder to measure cost per solved task alongside price per token. Sol does not erase the other frontier models. [Fable 5](https://www.coderabbit.ai/blog/fable-5-model-review) still feels stronger when you want architectural judgment or planning taste. [Sonnet 5](https://www.coderabbit.ai/blog/claude-sonnet-5-review) still has a cleaner comment-quality story in some review workflows. Sol wins on a more practical axis: you can hand it work and expect it to keep pushing. That is the switching question for this release. Do you need the model that sounds the smartest, or the model that finishes more of the work? ## **What's new in GPT-5.6** OpenAI describes GPT-5.6 as a family with capability tiers. Sol is the flagship model. Terra is the lower-cost option. Luna is the fastest and lowest-cost tier. The important change is that you can route work by depth now instead of treating GPT-5.6 as one model. The release changes four things for engineering teams: 1. Sol gives you a stronger long-horizon coding lane. It is the model to use when the task needs persistence across files, tests, and follow-up fixes. 2. Terra gives you a cheaper path for scoped implementation and first-pass review. 3. Luna gives you a low-reasoning lane for high-volume work where speed and cost are the main constraints. 4. Prompt caching is more predictable, with explicit cache breakpoints and a 30-minute minimum cache life. The price table also makes the competitive shape clearer. Sol is priced below Claude Fable 5 and close to Opus 4.8 on input price, while Terra and Luna create cheaper routing lanes for lighter work. | Model | Input price per 1M tokens | Output price per 1M tokens | | ----- | ----- | ----- | | GPT-5.6 Sol | $5.00 | $30.00 | | GPT-5.6 Terra | $2.50 | $15.00 | | GPT-5.6 Luna | $1.00 | $6.00 | | Claude Fable 5 | $10.00 | $50.00 | | Claude Opus 4.8 | $5.00 | $25.00 | ![Bar chart comparing API input and output token pricing for five different AI models.](/content/assets/image3-4.png) OpenAI also added more predictable prompt caching, including explicit cache breakpoints and a 30-minute minimum cache life. For long agent jobs, that can change how expensive the models can be. A model that reads the same repository for hours looks different once cache reads are working. Here is the working map: | Tier | Where we would use it | | ----- | ----- | | Sol | Long-horizon coding, harder review passes, multi-file implementation, Codex runs where completion is the goal. | | Terra | First-pass implementation, review triage, scoped fixes with escalation available. | | Luna | Low-reasoning work such as quick summaries, simple code explanations, PR summaries, lightweight review prechecks, test-name generation, and changelog scaffolds. | Luna is the tier to watch for top-of-funnel agent work. Use it where a cheap first pass has value on its own or can reduce the load before a Terra or Sol escalation. The key is low reasoning. If the task is mostly summarizing, labeling, extracting, or drafting a scaffold, Luna is the place to start. ## **The clearest signal was follow-through** One task made the pattern easy to see. The agent had to add stepped slicing to a programming language. That meant parser support, runtime behavior for arrays and strings, assignment behavior, Unicode correctness, and exact error messages. A weaker agent can make that look done. It edits the parser, runs one happy-path test, and stops. Sol did the useful version of the job. It inspected the parser and evaluator, added focused tests, handled edge cases like zero-length assignments and Unicode rune counts, ran the core suites, noticed an unrelated `go vet` warning, and submitted after verification. The process carried the signal. Sol did the dull work around the feature. We saw the same pattern in the live puzzle testing from the transcript. On the three-student word puzzle, Sol split the candidate words into letters and reasoned through each student's knowledge state. Terra reached the answer too, but its path looked more heuristic. The answer matched. The confidence did not. For coding agents, that gap changes how much you can trust the run. A model that guesses its way through a puzzle will guess inside your repo too. A model that builds a method and checks it has a better chance when the next task has no obvious path. Sol is best pictured as a persistent engineer: plain-spoken and stubborn about the list. I would still bring in another model for open-ended architecture. I would pick Sol when I need the boxes checked. We noted the same split in task terms. Fable tends to be better for architectural discussion, UI flow, and high-level judgment. Sol was stronger on lists, long-running implementation, existing code patterns, computer-use workflows, subagent coordination, and multi-day Codex runs. That is the useful distinction for teams. Use Fable when you want the smartest discussion. Use Sol when the queue needs to move. ## **Coding runs: Sol follows through** We also looked at a long-horizon coding run with more than 100 tasks across TypeScript, Go, Python, JavaScript, and Rust. Each task asks the agent to inspect a repository, change the code, and pass behavioral checks. The score is useful because it tracks completed software changes instead of answer style. ![Bar chart shows Sol model achieved 63.7% coding task pass rate, Terra 48.7%.](/content/assets/gpt-5-6-sol-and-terra-benchmark-media-03.png) | Model run | Pass rate | Completed trials | Average output tokens per task | | ----- | ----- | ----- | ----- | | Sol | 63.7% | 100% | 20,968 | | Terra | 40.7% | 100% | 55,594 | Sol passed 63.7% of tasks with no trial errors. That matches the hands-on experience: it stays oriented, keeps checking requirements, and works through the unglamorous parts of a repo task. Terra passed 40.7% of tasks. It also used more output tokens. Terra costs half as much per output token as Sol, while this run averaged 55,594 output tokens per completed task versus 20,968 for Sol. ![Bar chart compares output tokens and cost for Sol and Tera AI coding models.](/content/assets/gpt-5-6-sol-and-terra-benchmark-media-04.png) Terra may be cheaper for bounded work. For long coding jobs, measure cost per solved task before routing large volumes to it. ## **What this means for agent loops** The coding results point to a specific routing pattern. Sol should not be treated as a replacement for every model in the SDLC. It is strongest once the work has enough shape for an agent to execute, test, and improve. If you split code generation into an agent loop, I would route the work this way: | SDLC stage | First model to try | Why | | ----- | ----- | ----- | | Planning | Fable 5 for architecture, Sol for execution plans | Fable is better for open-ended tradeoffs. Sol is better when the plan needs to become a checklist the agent will actually follow. | | Research | Sol | It is strong at reading existing code patterns, tracing dependencies, and staying oriented across a large repo. | | Execution | Sol | This is the clearest fit. Sol is persistent, follows long task lists, and keeps working through the boring parts of implementation. | | Testing | Sol | It is better suited to running suites, interpreting failures, adding targeted tests, and looping until the patch holds. | | Improving | Sol for recall, Sonnet 5 for comment quality, Terra for cheaper triage | Sol finds more issues. Sonnet 5 is useful when developer attention is the constraint. Terra can screen lower-risk work before escalation. | The main change is where you can trust the agent to keep going. Planning still benefits from a second model when the problem is ambiguous. Once the plan is clear, Sol is the model I would put in the loop for implementation, test repair, and follow-up fixes. ## **Code review: Sol finds more, then filtering has to earn trust** Code review has a different bar. A coding agent can be verbose if it lands the patch. A reviewer has to catch the right issue and explain it in a way a developer will act on. We ran the new models through CodeRabbit's review benchmarks, using production-style pull requests with known issues. The goal was to see what happens when Sol or Terra is added in the review system. The baseline is our current ensemble. The delta shows the change when the model is added to that ensemble. A positive delta means the model found issues the current ensemble was more likely to miss. A negative delta means the model did not add enough useful signal in that run. The table uses six metrics: * `Pass` counts expected issues found in a way a developer could act on. * `Delta vs baseline` is the lift or drop after adding the model lane to the current ensemble. * `Pass full` gives broader credit for valid findings that may still need tighter wording or product filtering. * `Precision` is the share of actionable comments that were correct enough to keep. * `Comments` is raw model comment volume before product filtering. * `Nitpicks` are low-priority comments that can create review noise. Sol moved in the right direction for recall. In our internal review discussion, it found roughly eight more bugs than the baseline. The dashboard data shows Sol at 69 of 99 actionable passes, or 69.7%, with a \+7.4 point lift after adding the model lane to the ensemble. ![Code review performance charts comparing pass rates and actionable precision for Sol and Terra models.](/content/assets/gpt-5-6-sol-and-terra-benchmark-media-02.png) | Model lane | Pass | Delta vs baseline | Pass full | Precision | Comments | Nitpicks | | ----- | ----- | ----- | ----- | ----- | ----- | ----- | | Sol | 69 of 99, 69.7% | \+7.4pp | 74 of 99, 74.7% | 31.6% | 231 | 61 | | Terra | 53 of 101, 52.5% | \-8.6pp | 58 of 101, 57.4% | 35.7% | 143 | 21 | Sol is the stronger candidate for the main review pass. It is more critical and more complete, and it is less likely to wave a risky diff through. The cost is precision. Its actionable precision was 31.6%, down 8.2 points versus the baseline in that report, and it produced 231 raw comments. That tradeoff is workable in CodeRabbit because filtering is easier than recovering a missed bug. If the model sees more valid issues, the product can suppress weak comments, tune presentation, and route only the useful findings to developers. A model that never spots the issue gives you less to work with. Terra is the quieter lane. It produced 143 comments and 21 nitpicks, much lower than Sol. Its pass rate was also lower at 52.5%, with an 8.6 point drop versus the baseline average in that report. Its best category was Logic Error at 20 of 33, or 60.6%. It also produced 29 critical comments, 64 major comments, 50 minor comments, and only 9 outside-diff comments. ![Stacked bar chart showing review comment severity for Sai and Terra, dominated by major findings.](/content/assets/gpt-5-6-sol-and-terra-benchmark-media.png) I would test Terra for triage and cheaper review surfaces before I trusted it with the final pass on high-risk changes. The data points to restraint rather than strength. ## **How Sol compares to Fable 5 and Sonnet 5** The comparison needs one caveat: these numbers come from different harnesses. Our coding-run results for Sol and Terra are new in this review. The public Fable 5 and Sonnet 5 posts used code-review and coding-task evidence with different setups and denominators. Fable 5 was the autonomy model. In our Fable 5 review, it stayed close to the baseline on coverage, with 65 of 105 actionable review tasks versus 66 of 105 for the baseline and Opus 4.8. It reached 74 of 105 full review-task passes, slightly ahead of the baseline at 72 of 105\. The concern was precision and volume: 32.8% actionable precision, 19.4% full precision, and 253 comments. That made Fable 5 compelling for autonomous coding projects and harder to recommend as the default production reviewer. It could explore, plan, and build. It also ran long and produced enough review noise to make rollout harder. Sol feels more practical for execution-heavy coding work. Our findings point to the same split we noted above: Fable is better when the task benefits from taste or high-level judgment. Sol is better when you give it a long list and want every item handled. Sonnet 5 tells a different story. In our Sonnet 5 review, precision rose from about 29% on Sonnet 4.6 to roughly 38% to 40%. The catch was strict bug-catching. The baseline caught about 57%, Sonnet 5 landed around 50% to 51%, and Sonnet 4.6 caught about 63% while creating more noise. That makes Sonnet 5 a good fit when comment quality and developer attention are the main concern. Sol is more recall-first in our current read. It finds more, comments more, and needs product filtering around the output. ![Bubble chart showing code review model coverage, precision, and user comment volume.](/content/assets/image1-2.png) The routing guide looks like this: | Job | Best first pick | | ----- | ----- | | Architectural discussion | Fable 5 | | Long implementation run | Sol | | Review pass where missing the bug is costly | Sol, with filtering | | Review pass where comment quality is the main concern | Sonnet 5 | | Scoped repeatable work at lower unit price | Terra | The frontier models now feel different enough that one default model leaves useful capability on the table. ## **What Sol costs you** Sol's main weakness is fuzzy judgment. For architectural debate, product tradeoffs, or a plan with several defensible paths, I would still run Fable or Sonnet in parallel. Sol is strongest once the work has a clear shape. It can also get stuck. In our findings, one simple change took eight turns because Sol fell into an unhelpful path. We saw fewer of those moments than with GPT-5.5, but they still happen. Use stop points, checkpoints, and a second review thread for multi-hour or multi-day runs. Review trust is the other cost. Sol finds more issues, but developers only see the comments that survive the product. A raw model that posts too much can train people to ignore it. The review product still has to rank, filter, and explain. Terra has its own cost question. Its list price is lower, but the long coding run used more output tokens than Sol. For high-volume jobs, run your own cost-per-resolution test before moving large traffic. Cheaper tokens do not always mean cheaper solved tasks. ## **Should you switch?** Switch to Sol for coding agents if you are on GPT-5.5 or a previous OpenAI model. The long coding run is strong, and the hands-on behavior matches the score. Sol stays oriented longer, follows more requirements, and does more of the unglamorous work that makes an agent useful. Use Terra as a second lane. It is worth testing for scoped coding tasks, first-pass review, and workflows where escalation to Sol is available. Watch the total output bill. Use Luna as a first-pass lane. Save model-switch decisions for Sol and Terra, and route bigger tasks upward when speed and cost stop being the main constraint. Keep Fable and Sonnet in your toolkit. Fable remains the stronger pick for architecture and higher-level planning. Sonnet 5 remains attractive when you want cleaner review comments. Sol is the model I would reach for when I need the work finished. For most engineering teams, the switch path is clear: start with Sol on long-running implementation and harder review jobs, add Terra where the task is bounded, and keep your current best planning model for the parts where judgment needs more finesse.

The case for monorepos, and what they cost you

Santosh Yadav — Tue, 07 Jul 2026 00:00:00 GMT

When several teams depend on the same code, a design system, a shared API, and a set of business rules, keeping those copies in sync across separate repositories is time consuming. A monorepo puts that shared code in one place, so a fix lands once and a breaking change fails at review instead of failing silently in another team's service three weeks later. Once every team commits to the same tree, though, a bad change is everyone's problem; and code review becomes the thing standing between one careless PR and every downstream consumer of that code. I work as a developer advocate at CodeRabbit, and before that I spent years helping organizations adopt monorepos and scale the ones they already had, including a large migration at Celonis. This piece covers when a monorepo is worth it, the tooling that makes it practical, and how CodeRabbit keeps one reviewable once you've committed to it. ## Monorepo, polyrepo, monolith ![Monorepo diagram showing Analytics App, Design System, and Product Microservice connections.](/content/assets/image2-2.png) In a monorepo, every project lives in one repository. They can use different technologies and may or may not share code. This is what Google, Meta, and Microsoft use, and open-source tooling has made the same setup practical for much smaller teams. It's also the model behind the migration I'll keep coming back to in this post. ![Diagram showing a polyrepo architecture with various services and their corresponding repositories.](/content/assets/image1-3.png) A polyrepo gives each team its own repository, so instead of one repo with a hundred projects, you have a hundred repos, each owned independently. A monolith is a different kind of thing entirely, an architecture rather than a repository layout, an application built, deployed, and tested as a single unit. You can run a monolith inside a monorepo or split one across a polyrepo, since the word describes how software is deployed, not where the code lives. Polyrepo earns its place when teams are truly independent and can deploy without touching anyone else. The failure mode is the distributed monolith, separate repositories that still have to deploy together, which gives you the coordination cost of a monorepo with none of the benefits. The objection that kept smaller teams away from monorepos for years was build times. A monorepo rebuilds everything a change touches, so editing one piece of shared logic can mean rebuilding the design system, the web app, and the mobile app, even if only one of them uses your change. Build caching settled that. Free tools like Nx and Turborepo skip any project with inputs that didn't change, so a build only touches what you actually modified. You'll need somewhere to store the cache, and GitHub Actions gives you 10 GB per repository before you have to pay an S3 bucket or a remote-cache service. The build-speed problem that used to require a dedicated platform team is now a config decision. ## When a monorepo is the right call The deciding question is whether your teams actually share code. If your projects are independent and deploy on their own, polyrepo is simpler. If several teams maintain the same design system, business logic, or APIs, a monorepo is what keeps everything in sync. The pattern usually starts innocently. A team ships something standalone, then it grows and starts sharing a design system or a microservice with other apps. On paper, everything stays independent. But the bill arrives later when you're maintaining several live versions of the same shared code, and every upgrade means building and publishing them all. A monorepo also wins when you need atomic changes across services. When you introduce a breaking change to a microservice that every app depends on, a polyrepo scatters the fix across repositories, and rolling it back after a production break becomes a coordination problem involving every affected team. In a monorepo, the same change lands as a single commit with a single history: One revert undoes it everywhere. The same logic applies to large refactors, which is where polyrepos stall: you sit blocked, waiting for another team to ship the change you depend on. Over time this shapes culture too, nudging teams toward reusing shared code instead of reinventing the same utility five times. At a previous company, my team worked on one interconnected product, but under polyrepo every squad built its own frontend against a shared design system. Upgrading Angular meant chasing each team's repo one at a time, it was a miserable process. Over a couple of years' time, we consolidated everyone onto an Nx monorepo, and the change showed up in the numbers: Releases went from occasional to more than 40 apps a week. Angular upgrades and new tooling stopped being projects and became routine, and code sharing through internal libraries finally worked the way it was supposed to. None of this makes polyrepo wrong. It makes it the answer to a different question, and forcing unrelated projects into one repo causes its own problems. ## When polyrepo is the better choice Polyrepo has real uses, and one is worth stating up front. You can run several monorepos, so "not everything in one repo" doesn't mean "one repo per project." Keep things apart when teams own unrelated products that share no code and ship on their own cycles, where a monorepo buys you nothing and a separate repo keeps release timing and customer-facing changelogs clean. Compliance can make the split non-negotiable, since government contractors and customers who require that only named people touch the code are best served by a repository walled off from everything else. Open-sourcing is the other common case, where a dedicated public repo lets outside contributors in without dragging the rest of your codebase along. ## Keeping a monorepo healthy with CodeRabbit Let’s say you've made the call and moved to a monorepo. The bet comes due here. Every advantage came from putting teams in the same tree, and that same closeness means one careless PR now reaches everyone downstream. On a busy monorepo the reviewer catching it is a person working through dozens of PRs a week, and people miss things. On top of that sits a newer pressure: a growing share of the code in pull requests wasn't written by anyone on the team. In an [agentic SDLC](https://www.coderabbit.ai/guides/agentic-sdlc), a coding agent produced it, and the reviewer is now evaluating code they didn't author and may not fully understand. That changes the question from "would I have written it this way" to something harder: "what does this actually do, and where does it break." CodeRabbit's analysis of [AI-authored versus human-authored PRs](https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report) found that AI-generated changes carry roughly 1.7x more issues on average, and more than 3x more readability problems. In a monorepo, that code lands right next to everyone's shared logic, which makes the comprehension gap a shared problem too. CodeRabbit takes the repetitive first pass so your reviewers can spend their attention on architecture and intent. On a monorepo, a few things make it worth having: * It traces changes through the code graph and flags breaking API changes against the callers they affect, before the PR merges. * It opens each review with a plain-language summary, a walkthrough, and an architecture diagram, so a reviewer facing an unfamiliar change can see what it does before judging whether it's right. * It holds style and convention consistent across packages and calls out missing docs, which is where shared code rots fastest. * It runs 50-plus linters and SAST scanners inline, and learns from your team's feedback over time rather than applying a generic checklist. None of this replaces human review. It makes the human faster and keeps the standard, even across teams that would otherwise drift apart. The context isn't limited to a single tree either. For polyrepos, Linked Repositories and [Multi-Repo Analysis](https://docs.coderabbit.ai/knowledge-base/multi-repo-analysis) extend the same reasoning across related repositories, so a renamed field in an API schema surfaces against the services that parse it even when they live somewhere else. That's what matters for teams with microservices, shared SDKs, or a split frontend and backend.

Claude Sonnet 5 review: Should you switch?

Juan Pablo Flores — Tue, 30 Jun 2026 00:00:00 GMT

Anthropic just shipped [Claude Sonnet 5](https://www.anthropic.com/news/claude-sonnet-5), the newest model in its mid-tier line. This post answers one simple question: Should you stay on the model you use today, or move up to this one? Sonnet 5 didn't land out of nowhere. We've reviewed a string of new models on the blog over the past month: * [Opus 4.8](https://www.coderabbit.ai/blog/opus-4-8-release) was the best model we'd tested for long, multi-step coding, careful enough to follow review instructions to the letter, though it cost more and paid off mostly on bigger jobs. * [Fable 5](https://www.coderabbit.ai/blog/fable-5-model-review) leaned even harder into coding on its own, and was happy to plan and build across lots of files. However, its price and limited access kept it off our default review path. * NVIDIA's [Nemotron 3 Ultra](https://www.coderabbit.ai/blog/nemotron-3-ultra-release) went the other way: fast and to the point, an open model built to take lots of quick swings inside an agent setup. Sonnet 5 feels like the rest of the Anthropic family. It's patient and thorough, and it likes to think a problem all the way through before it acts. For writing and building code, Sonnet 5 is the most capable model we've worked with at this tier, and it's an easy upgrade to be excited about. For review, it's more of a tradeoff. While it generates cleaner, sharper comments, it catches fewer bugs than the earlier models we currently run in production, and comes at a slightly higher cost per review. The good part is that you can tune most of that, and for a lot of teams the move is well worth making. Let's walk though what's new in Sonnet 5, how it performs writing and reviewing code, and if it makes sense to switch. %[https://youtu.be/7XE846SJ84E] ## **What's new in Claude Sonnet 5** Sonnet 5 thinks more deeply than the version before it. For you, that means it works through harder problems the old model would have given up on. The thinking "effort" dial in Sonnet 5 that allows you to incrementally turn it down to off if you don't need it. This is the feature that protects your budget. Turn the effort up for a tricky review where a missed bug is expensive, and turn it down or off for routine work where you'd rather not pay for deep thinking. It can also rewrite its own instructions partway through a task. On long agent jobs, the goal tends to shift as the model learns more, and a model stuck on its first plan will keep pushing on instructions that no longer fit. Sonnet 5 updates its own plan instead, so you get fewer runs that wander off and burn your time and tokens. It also ships with new safety guardrails around security and cyber topics. The upside is fewer risky outputs. The catch is that real security work can trip the filters now and then, so expect the odd refusal if that's your area. ![Presentation slide detailing three key improvements of Sonnet 5 for code reviews.](/content/assets/sonnet-5-benefits.png) The easiest way to picture Sonnet 5 is as a mid-level engineer who cares, maybe a little too much, about shipping code that truly works and runs at the level you asked for. That one trait shapes most of what it does, and it showed up in four habits we saw again and again: * It tends to write tests before the feature. * It keeps polishing a solution long after it already works. * It second-guesses its own plan, sometimes more than necessary. * It answers a small task with a small project. ## **How Claude Sonnet 5 writes code** Sonnet 5’s code writing is the main reason we think most teams will want to upgrade. Before we get to the review numbers, it is helpful to see how it behaves when it builds something from scratch. We handed it the work we do daily, from quick features to harder problems with no clear path. One evening, we gave it a tough task and stepped out for lunch. The job asked the model to write the code, run simulations on it, and keep tuning the output until the results were as good as possible. We came back expecting it to be done or stuck. Neither were true. The model was still going, working through the problem on its own, with no nudging from us. It built the whole application by itself. The reason it ran so long was simple. It kept cleaning up its own solution, pass after pass, because it was chasing the best answer instead of the first one that happened to work. That's the kind of thing that was previously only available with a pricier model, and watching a mid-tier model do it is a real moment. That aforementioned diligence is exactly why Sonnet 5 is good at long, open-ended work. The slowness that bugs you on a one-line change turns into a strength when the job has no set number of steps. It's a great fit for agent loops, where you hand the model a simple goal and let it spend a few rounds trying approaches, testing them, and improving the result on its own before it reports back. Anthropic's own guide to [building effective agents](https://www.anthropic.com/research/building-effective-agents) calls this an evaluator loop, where the model writes a result and then critiques and improves it in a cycle. It's worth a read if you're building that kind of workflow. Sonnet 5 has that loop wired into how it thinks, and if you've been holding off on agentic coding because earlier models drifted or gave up, this is the release that changes that. ### **It treats testing as a habit** Most models treat tests as a chore for later, Sonnet 4.6 included. Sonnet 5 tends to write the tests first, builds the feature on top of them, and then runs everything once it thinks it's done. The self-checking we saw comes straight out of that order. You can't spot a clash between your tests and your code if you never run the tests, and this model always runs them. If you've ever shipped code that looked fine and broke a week later, you'll feel that difference fast. ### **What the extra care costs you** The extra care Sonnet 5 takes shows up in your file count and token use. Ask for something small and you get a lot back. You'll see extra helpers and a test file longer than the feature itself. On a big feature that looks like good engineering. On a one-line change it looks like a model that can't help itself. We found Sonnet 5 to be slower than Sonnet 4.6, which is likely due to the extra thinking Sonnet 5 performs. You're trading minutes for thoroughness, which pays off on long jobs you leave running. It does sting when you're waiting on a small change. Where 4.6 hands back a quick answer, 5 keeps working for a better one, and you feel that most on the little stuff. Two smaller notes: It uses more tokens than 4.6 did, and is still clear, just chattier. It also writes good plans, then rewrites them mid-task more often than we'd like. None of this is a dealbreaker for build work, but it's worth knowing before you point it at a pile of tiny edits. ## **How Claude Sonnet 5 reviews code** This is the part we care about most, since reviewing code at scale is what we do. AI now writes a big share of the code many teams ship, and that code needs a careful second pair of eyes. Our look at 470 open-source pull requests found that AI-co-authored PRs carry about [1.7 times more issues than human-only PRs](https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report), and one [2025 study](https://arxiv.org/html/2509.19708v1) found review time climbed 91% on teams leaning hard on AI. The model doing the reviewing is now a big factor in how fast teams ship. We added Sonnet 5 to our harness and pointed it at our standard benchmark, a fixed set of pull requests with known bugs, and measured how many it caught and how clean its comments were. You can read more about how we build these reviews in our piece on [context engineering for AI code reviews](https://www.coderabbit.ai/blog/context-engineering-ai-code-reviews). ![Recall vs. precision plot compares Sonnet 5 and 4.6; 4.6 catches more bugs.](/content/assets/image1-12.png) ### **The comments are much cleaner** Sonnet 5's comments are cleaner and its findings were more often bugs than noise. On our benchmark, precision climbed from about 29% with Sonnet 4.6 to roughly 38% to 40%. Sonnet 4.6 does the opposite. It comments on almost everything and leaves you to pick the keepers out of the clutter. If you've ever turned a reviewer off because it flagged every little thing, you already know how important it is to highlight what needs your attention.. ![Bar chart comparing Sonnet 5 and 4.6 precision for different comment types.](/content/assets/image2-8.png) ### **The honest catch on bug-catching** Inside our review harness, Sonnet 5 catches fewer bugs than our current production setup. On the strict "did it find the bug" measure, our baseline catches about 57%, and Sonnet 5 lands around 50 to 51%. What surprised us was that Sonnet 4.6 caught more than either of them, around 63%, even though it's a noisier reviewer. So the model that buries you in comments was also the one that missed the fewest bugs. Turning Sonnet 5's effort dial up barely moved its score and roughly doubled the cost. On a looser score that counts more types and variations of comments, high effort brought its findings back to about even with the current baseline, but it didn't pull clearly ahead. ![Bar chart compares Sonnet 5 and 4.6 bug catching; 4.6 significantly improves performance.](/content/assets/image4-3.png) There's a bit of noise hiding in the comments too. Sonnet 5 posts three to four times more nitpicks than our baseline and almost 80% more than Sonnet 4.6 in high effort runs. Its best comments read cleaner, but you have to look past more minor ones to find them. We also tried it with thinking turned off. On some jobs it matched our baseline, on others it slipped a little. So there's a class of lighter review work you can run cheap with thinking off and lose almost nothing, as long as it isn't your most important code. The takeaway isn't that Sonnet 5 is a weak reviewer. It's actually a quieter, more careful one, and for teams drowning in review noise, that's often the better trade. ## **Claude Sonnet 5 vs the model you run today** Reach for Sonnet 4.6 when raw coverage is what you need and you've got the bandwidth to wade through its comments. Reach for Sonnet 5 when you'd rather get fewer, sharper comments and a much stronger partner for actually writing code. Sonnet 4.6 finds a few more bugs, while Sonnet 5 wastes far less of your attention. ### **Sonnet 5 versus a flagship model** Against flagship models, like the Opus family that teams keep for their hardest jobs, Sonnet 5 does nearly as well on review quality for a lot less money. If you're paying flagship rates today only because nothing cheaper was good enough, Sonnet 5 is worth a look. Keep an eye on the token bill while you test, because a model that thinks this much can eat into its own savings. Cranking the effort to maximum was the worst deal of the bunch. It roughly doubled the cost without finding meaningfully more bugs, so don't reach for the top tier by default. We saw the same pattern when we reviewed [what Claude Opus 4.7 means for AI code review](https://www.coderabbit.ai/blog/claude-opus-4-7-for-ai-code-review). ## Claude Sonnet 4.6 or Claude Sonnet 5: Quick verdict For most teams, the answer is yes. Sonnet 5 is the most exciting coding model we've used in its class. It works like a careful teammate, the kind who would rather take an extra few minutes than hand you something that breaks later, and that instinct makes it a clear step up from 4.6 for anyone doing real building work. In plain terms: switch now if you write or ship real software and want a model that tests its own work and sticks with a hard problem until it's solved. Run it at medium effort and you get most of the upside without the top-tier price. And if you're paying flagship rates purely for quality, line Sonnet 5 up against your current model before the next renewal, since it may match the work for a fraction of the spend. The one group that should wait is high-volume teams with tight latency budgets and lots of tiny diffs. That's the workflow where its slower, more careful style doesn't earn its keep yet. It isn't perfect, and we've been upfront about where it lags on review. But its strengths in writing and reviewing code outweigh that easily. If you've been waiting for a reason to upgrade, this is it. Want the full picture on AI-generated code quality? Read our [State of AI vs Human Code Generation report](https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report), see [how teams are actually using AI dev tools](https://www.coderabbit.ai/blog/ai-adoption-how-developers-are-using-ai-dev-tools), or try CodeRabbit free [in your IDE](https://www.coderabbit.ai/ide).

Loop engineering: designing loops you can actually walk away from

Hendrik Krack — Thu, 25 Jun 2026 00:00:00 GMT

A new term is currently trending across X and capturing the attention of the engineering community: loop engineering. The core concept is straightforward: move beyond manual interactions with your coding agents. Instead, engineer autonomous loops that handle the execution for you, freeing you up to focus on other tasks, or perhaps design your next loop. Peter Steinberger (Creator of OpenClaw) [kicked off the debate](https://x.com/steipete/status/2063697162748260627?s=20\() , and Boris (Head of Claude Code) [famously quoted](https://x.com/rohanpaul_ai/status/2063289804708835412) “I’ve moved past manual prompting. Now, I design autonomous cycles that direct Claude and navigate the execution path for me. My real work is engineering the loops themselves.” The industry shifted from prompt engineering in early 2023 to context management, and eventually to harness engineering, which gained traction late in 2025. By the time 2026 was being hailed as "the year of agent harnesses," the focus had already pivoted once more. For the broader production workflow around these systems, see [the engineer's guide to a coding agent workflow](https://www.coderabbit.ai/guides/coding-agent-workflow). ![Four steps showing the evolution of engineering: Prompt, Context, Harness, and Loop engineering stages.](/content/assets/image4.png) ## Where loop engineering started The idea isn't even new. Back in January, [Geoffrey Huntley](https://x.com/GeoffreyHuntley) was already planting the flag with what he calls the [*ralph loop*](https://ghuntley.com/loop/) \- feed an agent the same goal over and over, let it run for hours, and have it find, plan, and resolve the work on its own with nobody in the seat. Rather than relying on the model's internal context, progress is preserved through git history, files, and external memory management, which remains the primary technical hurdle. By utilizing a basic bash script, a new agent can seamlessly resume the task exactly where its predecessor stopped once the context window is exhausted. Anthropic published [the same shape](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents) from the research side a couple of months earlier: an agent working in shifts, each one showing up with no memory of the last, leaning on notes left on disk to carry on. ## What does a loop look like? While prompt, context, and harness engineering all required you to remain hands-on manually guiding the agent through every turn, loop engineering represents a total shift. It allows you to step away completely by designing a system that operates autonomously, removing the human from the loop entirely. The fundamental distinction lies in who is in control: * **Prompts** serve as isolated instructions; the AI provides a single response and then halts, waiting for your next input. * **Loops** are recursive goals. Once the purpose is established, the system independently navigates the entire circuit, persisting until the objective is fully realized. ![A circular diagram illustrates a five-step iterative process called The Loop.](/content/assets/image2-4.png) ## What are the paradigms needed for a loop? Regarding the actual architecture, Addy Osmani provides the most streamlined breakdown: [five core building blocks](https://addyosmani.com/blog/loop-engineering/) plus a dedicated memory layer. While terminology may vary across different platforms, the underlying functionality remains consistent. * **Automations**: Something on a schedule that hunts down work and triages it before you've had your coffee. * **Worktrees**: Parallel agents each have their own checkout, so two of them can't overwrite each other's files. * **Skills**: Your project's knowledge is written down once: conventions, build steps, the hard-won "don't do it this way" notes, so the agent stops re-guessing every session. * **Plugins and connectors**: Built on MCP, this is how the loop reaches past the filesystem into the tools you actually use: your issue tracker system, a database, staging environment, Slack. * **Sub-agents**: The model that wrote the code isn't the one that grades it, because it's a soft grader of its own work. * **State**: The sixth piece, and the easy one to underrate, memory on disk. A file or a Linear board that tracks what's done and what's next, because the model resets between runs even when the repo doesn't. ## How do you design a successful full loop? Back in February I wired up a loop for a personal project, mostly to see how far it could run without me. It started with real user feedback, as the loop pulled in new requests, triaged them, and drafted a plan for each one worth doing. From there Claude wrote the implementation, CodeRabbit reviewed it in a loop until the diff came back clean, and Claude ran the tests. If everything held, the loop waited for CI, merged on its own, and then re-verified the result with a post-merge deploy check. My whole job was deciding what was worth implementing and verifying what the agents produced. I designed the loop once, and it kept shipping. ![Software dashboard showing a dark-themed pipeline status with phases, runs, and failure signatures.](/content/assets/image1-18.png) What actually held it together was boring, and that's the point. The whole thing ran off one scheduled job and a state file: a plain markdown log of what shipped, what failed, and what was still open, so each run resumed instead of starting cold. The project's conventions lived in a skill the agents read every pass, so nobody re-derived the setup from scratch. And the part that let me leave it alone was the quality gate. Nothing merged unless the tests passed and CodeRabbit's review came back clean, which meant "done" was a signal I could trust, not Claude's opinion of its own work. I kept it on small, checkable changes, the kind a careful junior could ship from a ticket, and left anything that needed real judgment for myself. The loop didn't make those calls, it just made the calls I'd already encoded, over and over, without me. For a deeper version of that gate, see [building a quality gate that works for AI-generated code](https://www.coderabbit.ai/guides/ai-generated-code-quality-gate). ## How CodeRabbit fits into loop engineering On the CodeRabbit side it was three pieces working with Claude Code: the planning agent turned raw feedback into a coding plan, the CLI ran reviews right inside the loop so Claude could fix findings before anything became a PR, and the review product was the final gate on the PR itself. Here's the shape of it: ![CodeRabbit workflow diagram showing plan, build, review, gate, ship stages with a re-review loop.](/content/assets/image3-8.png) ## To build or to not build a loop Before you build one, ask whether the task is even worth it. Loops reward a stable target: rewrite a codebase and the goal holds still, so one verifier (does it build, does the tests pass, does behavior still match) carries every iteration, written once and reused the whole way. But when the conditions keep shifting, the math flips. If each run needs its own definition of "done," you spend your time rewriting the verifier instead of shipping, and that upkeep eats whatever the loop was supposed to save. Stable goal, build the loop. Moving target, keep it a manual prompt.

How a hackathon project turned into my work at CodeRabbit

Ayush Sridhar — Mon, 22 Jun 2026 00:00:00 GMT

In late October, I visited CodeRabbit’s offices to collect my prize from the CalHacks Hackathon. A few months later, I walked into the CodeRabbit office as a SWE intern. That’s the headline. But the part that matters (especially if you’re a junior dev) isn’t “wow, lucky.” It’s the chain of events that made it possible: ![A four-step process: Build something real, Ask for help, Iterate fast, Follow up with proof.](/content/assets/image1-7.png) This post is for junior developers who are trying to land internships in a market that feels tougher than it should be. I want to share the “unofficial path” that worked for me. The technical project that started it all was: **merj**, an AI-powered merge conflict resolver that used CodeRabbit to give an LLM the missing ingredient it needed to resolve conflicts correctly: semantic intent. %[https://youtube.com/shorts/6chYSna0_QU?si=QooHLMvDyQEvXmHm] I’ll also share what I’m working on now as an intern. I’m taking the same core idea and turning it into something that can help teams resolve PR merge conflicts with a workflow that fits cleanly into GitHub. ## **October: Hackathon chaos to hackathon WIN** CalHacks is chaotic and energetic in the best way. There’s half-coded demos, half-asleep teams, and a constant stream of “that doesn’t work, gotta fix that.” My friends and I did what a lot of teams do at the start. We brainstormed a bunch of ideas, then walked around talking to sponsors and attendees to see what’s feasible and what would actually be fun to build in a weekend. That’s when we ran into CodeRabbit. At the time, we weren’t thinking “we need a code review product.” We were thinking like hackers: “We want to ship something ambitious in 36 hours and how do we keep it from collapsing under rushed changes?” We realized CodeRabbit could help. And that’s when we met [Hendrik Krack](https://www.linkedin.com/in/climateadvocateaienthusiast/)**,** a developer advocate at CodeRabbit. This is the moment that changed everything. Hendrik didn’t just tell us about the product. He helped us refine the actual project idea in a way that made CodeRabbit genuinely *useful* to the solution. Our early thinking was pretty typical hackathon-thinking: “Throw some context into an LLM and hope it figures it out.” Hendrik pushed us toward a much better approach: “Don’t just give the model raw diffs. Give it semantic understanding of what each person was trying to do.” That might sound obvious in hindsight. But in the middle of a hackathon, it’s easy to default to “LLM magic.” Hendrik helped us see what the LLM would actually *need* to succeed and that became the core of our architecture. Over the course of the hackathon we stayed in contact with Hendrik and also met [Erik Thorelli](https://www.linkedin.com/in/erikthorelli/), the Dev Experience Lead at CodeRabbit. We attended their workshop, asked for input multiple times, and kept tightening our ideas based on what we learned. ![Five men stand by a large window overlooking a body of water and the distant horizon.](/content/assets/img-4210.jpg) By the time presentations came around, we were showing something that had been iterated on with real feedback, and we even got the chance to present directly to them because they were so interested in what we were building. We ended up winning CodeRabbit’s track. The prizes were amazing: Meta AI glasses and a visit to the CodeRabbit office that afternoon. But the part that stuck with me was the irony. I visited the office as a hackathon prize, and I had no idea I’d be working as an intern in that same office less than three months later. ## **What we built at CalHacks: merj, an AI-powered merge conflict resolver** Our project was called **merj** (pronounced like “merge,” but slightly wrong on purpose). **Problem:** Merge conflicts are one of the most tedious, annoying parts of software development. They show up at the worst times. They’re hard to reason about from conflict markers alone, and they’re a perfect example of “high friction, low reward” work. You’re not building something new. You’re just trying to get back to building. So we asked: What if resolving a merge conflict could be as easy as running one command? ### **The user experience** The tool was a CLI, and the entry point was simple: `merge pull` %[https://www.youtube.com/watch?v=R1SPVzI1epU] Under the hood, it would attempt a normal `git pull`. If Git complained about a merge conflict, merj would trigger a conflict-resolution workflow. We designed it that way intentionally. We didn’t want “another AI chat tool.” We wanted something that fit the existing workflow developers already use. ## **The part that mattered most (and it wasn’t the prize)** Over that weekend, Hendrik helped us refine our concept so we weren’t relying on “LLM magic.” He showed us how to use CodeRabbit’s semantic summaries as an intent layer, and he was generous with his time answering our questions about how CodeRabbit works and how it could integrate into our project. Erik’s workshop helped us understand CodeRabbit more deeply so our integration became less hand-wavy and more real. We kept coming back for feedback, and each time, the project got better. This is one of the most important lessons I’ve taken away as a junior dev: **Asking for help is a skill. And when you do it well, with momentum and clarity, experienced engineers actually want to help.** A good “ask for help” looks like: * “Here’s the goal.” * “Here’s what we built so far.” * “Here’s what isn’t working.” * “Here are two options we’re debating.” * “What would you do next?” That’s how you turn a sponsor booth conversation into a real connection. ## **The follow-up: How a hackathon turned into an internship** The week after CalHacks, I reached out to Hendrik and Erik asking about potential internship opportunities and whether they could help me navigate the process. They connected me with David Loker, CodeRabbit’s VP of Applied AI, and we set up an interview, which felt like a continuation of the hackathon: * I thoroughly explained our project architecture. * We talked about follow-ups and how to improve the product. * We discussed how it might fit within CodeRabbit’s existing product. * We went deeper on technical choices and failure modes. I received an offer the next day. I’m not sharing this to flex. I’m sharing it because it highlights something that matters in today’s environment: It’s easier for people to believe in you when you’ve already shown proof of how you think, build, and iterate. ## **Practical advice for junior devs (what I’d do again)** I want to end with concrete advice, because I wish someone had told me this earlier. ### **1. Build things that force real engineering decisions** Tutorials are fine. But the projects that change your trajectory are the ones with: * Constraints * Tradeoffs * Failure modes * Real integration points Those are the projects that make interviews feel like conversations instead of interrogations. ### **2. Ask for help while you’re still building** Don’t wait until it’s “perfect.” Ask early enough that feedback can change the design. ### **3. Make your ask easy to answer** Respecting someone’s time isn’t about being silent; it’s about being clear. Show: * What you tried * What broke * What you think is happening * What kind of feedback you want ### **4. Follow up with proof, not desperation** “Here’s what I built and what I learned” is memorable. A generic “please give me a chance” message isn’t. ### **5. LeetCode can help you pass gates. Building creates doors.** Interview prep matters, but building impressive things is what gets attention and creates opportunities. ## **The bottom line** I landed an internship with CodeRabbit because I built something real, asked for help early, iterated fast, and followed up with proof. And the wild part is that the loop is continuing. The hackathon project that started as a weekend experiment is now becoming a real product initiative I get to work on as an intern. Don’t just apply harder in isolation if you’re a junior dev trying to stand out in a tough environment. Instead, build something you’re proud of and then get it in front of people who might be interested in it.

Automatic repository linking: cross-repo context without manual setup

Erfan Al-Hossami — Thu, 11 Jun 2026 00:00:00 GMT

A pull request can look safe in one repository and still break another service that depends on it. That is why we shipped [Multi-Repo Analysis](https://www.coderabbit.ai/blog/Coderabbit-multi-repo-analysis): to help CodeRabbit review changes with context from related repositories. The challenge was set up. Teams still had to tell CodeRabbit which repositories were connected, and that list could go stale as services, packages, and ownership changed. [Automatic Repository Linking](http://docs.coderabbit.ai/knowledge-base/automatic-repository-linking) removes that manual step. CodeRabbit can now detect related repositories across your organization and use them as review context, so cross-repo impact is easier to catch before you merge. %[https://youtu.be/cKE6q5JLQ8Y] ## What Automatic Repository Linking does Automatic Repository Linking discovers and links related repositories across your organization. It first analyzes signals that define a dependency such as import graphs, dependency manifests, and shared code patterns. It then links the repositories that depend on each other, so CodeRabbit knows your architecture without anyone maintaining a list. What that gives your team: * **Cross-repo context with zero setup.** Related repositories get linked automatically. You do not need to maintain a static list of related repositories as services and packages evolve. * **Links that reflect how your code fits together.** Because the connections come from the code itself, they map to real technical dependencies and stay current as your services evolve. * **The same reviews, on more of your code.** Auto-discovered links feed directly into CodeRabbit's research agent, which explores linked repositories in real time and surfaces breaking changes with exact files and line numbers before you merge. Auto-linked repositories are stored separately from manual links. Manual links are not overwritten. Automatic Repository Linking also builds on CodeRabbit’s knowledge base. If your organization has opted out of knowledge base features, both Multi-Repo Analysis and Automatic Repository Linking is disabled. ## How it works and why our approach is different ![Diagram showing CodeRabbit's automatic repository linking, connecting a service to related repositories.](/content/assets/image1-5.png) Cross-repo context has become a useful feature for coding agents and code reviewers, yet most approaches still depend on a human to draw the map. Some tools widen their view beyond a single repository only when you scope that context by hand, whether by passing extra directories to the agent, opening a multi-root workspace, or declaring related repositories in a configuration or agents file. Others automate the discovery step but infer relationships from team activity, such as which contributors have recently committed where. That can show how people work, yet it does not prove that two repositories are actually related. Distinctly, Automatic Repository Linking starts from code-level evidence. CodeRabbit looks across eligible repositories in your organization for signals such as imports, dependency manifests, API usage patterns, contracts, and repository README files. When those signals point to a real relationship, CodeRabbit links the repository as review context. During review, CodeRabbit combines those auto-detected links with any repositories your team already configured manually. The multi-repo research agent can then inspect the linked repositories and surface downstream impact with concrete files and lines. The Review info section shows which repositories were considered. ## Get started %[https://youtu.be/ckKYd1-e1CI] You should not have to keep a manual map of every service, package, and repository relationship in your organization. With Automatic Repository Linking, CodeRabbit keeps that context closer to the code, so reviews can catch downstream impact before it reaches production. Automatic Repository Linking is available for Pro plus and Enterprise customers. Enable it in your knowledge base settings and start sharing context between your repositories. **Learn more** * [Introducing Multi-Repo Analysis](https://www.coderabbit.ai/blog/Coderabbit-multi-repo-analysis) * [Multi-Repo Analysis documentation](https://docs.coderabbit.ai/knowledge-base/multi-repo-analysis)

Humans don’t have an API

David Kravets — Wed, 10 Jun 2026 00:00:00 GMT

Are we treating coworkers like AI agents? Take a look at your recent Slack messages, Google Docs comments, emails, or the transcripts of your last few video calls with colleagues. How many of them began with a greeting, or provided context for a sudden request? Now look at your chat history with your preferred AI assistant. One of the more curious features of the modern workplace is that these two columns of text can sometimes look surprisingly similar. As generative AI becomes embedded in daily work, the line between how we communicate with software and how we communicate with one another can feel less distinct than it once did. Direct requests, immediate responses, and highly task-focused exchanges have become a routine part of interacting with AI systems. As organizations optimize for efficiency, it is worth considering how those habits may influence communication in the workplace as well. ## **The side effects of talking to machines** A common response to concerns about AI etiquette is straightforward. The system has no feelings, so how we speak to it doesn't matter. However, behavioral scientists have long understood that habits formed in one context often spill over into others. So, does spending hours each day issuing commands to conversational AI systems change how we communicate when another human being is on the receiving end? The evidence is still emerging. However, new research suggests that prolonged interaction with AI systems may influence interpersonal communication styles in subtle but meaningful ways. Generative AI rewards directness. It responds instantly, stays focused on the requested task (unless there’s hallucinations) and converts instructions into results with remarkable speed. The interaction is efficient, goal-oriented, and largely free of the social rituals that characterize human conversation. Over time, it is reasonable to wonder whether those expectations begin migrating into our human relationships as well. Human relationships are built through repeated interactions that create trust, understanding, and shared context over time. These moments may appear inefficient when measured purely by output, yet they help build the goodwill and mutual confidence that make productive collaboration possible. When we begin treating coworkers like conversational interfaces, like APIs, those relational investments become easier to skip. ## **What some emerging literature says** Because generative AI arrived so quickly, research on its social consequences is still catching up to the reality that millions of people are spending hours each day interacting with machines capable of producing human-like conversation. Nevertheless, research is beginning to examine how those interactions may influence human relationships and workplace behavior. ### **The AI "social forcefield"** Researchers Christoph Riedl, Saiph Savage, and Josie Zvelebilova explored this phenomenon in their paper [*Cognitive Spillover in Human-AI Teams*](https://arxiv.org/abs/2407.17489). The researchers conducted two randomized experiments to examine whether interactions with AI influence subsequent human-to-human communication. Across both experiments, they found evidence of what they call "cognitive spillover," where the effects of AI exposure carried forward into later interactions between people. According to the authors, AI exposure influenced shared language, collective attention, shared mental models, and social cohesion. The researchers describe this phenomenon as an "AI social forcefield." The term reflects the paper's central argument that AI shapes the social and cognitive environment in which collaboration occurs. In the researchers' framing, AI functions as part of the environment that influences how people communicate, coordinate, and develop shared understanding. Their findings suggest that AI influences more than the quality or speed of work. It can also shape how people focus their attention, exchange information, and build common ground with one another. The paper focuses on controlled experiments rather than long-term workplace behavior. Its findings nevertheless raise an important question for organizations rapidly adopting AI. If interactions with AI can influence subsequent human-to-human communication, what happens when those interactions become a daily part of work? ### **A warning flare about human relationships** If the cognitive spillover research offers evidence that AI can influence human communication, another paper raises a broader concern about where those effects might lead. In [*Chatbots and Human-Human Relationships: The Need for Research on Potential Downstream Harms from Generative AI*](https://www.tandfonline.com/doi/full/10.1080/13668803.2026.2623500?), researchers Justin Keeler and Brett Murphy issue what amounts to a warning flare. Their central argument is that society is adopting conversational AI systems far faster than researchers understand their long-term effects on human relationships. Rather than presenting experimental findings, the paper identifies a set of potential downstream harms that the authors believe warrant further study. Among the concerns they discuss are reduced social interaction between people, spillover effects from chatbot interactions into human relationships, and the possible erosion of social abilities. A central theme of the paper is reciprocity. The authors note that conversational systems can provide relational benefits to users without requiring reciprocal efforts in return. Human relationships operate differently. They depend on mutual obligation, compromise, empathy, and ongoing investment from both participants. The researchers argue that this difference raises important questions about how widespread interactions with conversational AI may influence human relationships over time. The paper presents these concerns as hypotheses requiring further investigation rather than established conclusions. Its purpose is to encourage debate, not settle it. The underlying question is difficult to ignore. How might communication habits developed through interactions with conversational AI systems influence the way people relate to one another? ## **The parts of work that don’t scale** Both papers point in the same direction. One provides evidence that interactions with AI can influence subsequent human communication. The other argues that society has only begun to explore the long-term consequences of those influences. If AI changes how people relate to one another, what exactly is at stake? The answer begins with the value of human connection itself. Decades of workplace research have found that employees who feel supported, valued, and connected to the people around them are more engaged in their work. Organizations thrive on more than the exchange of information. Trust, cooperation, mentorship, and shared purpose shape how work gets done. Workplaces can automate tasks. Relationships still have to be earned. ## **Humans don't have an API** AI systems are designed to transform prompts into responses. A prompt arrives, the system processes it, and a response follows. The interaction is immediate, task-oriented, and highly predictable. Human collaboration operates differently. Colleagues bring experience, judgment, competing priorities, emotions, relationships, and context to every interaction. A request often becomes a conversation. Discussions lead to new ideas. Detours can reveal better solutions than the one originally imagined. Beyond exchanging information, the strongest teams develop common ground, challenge assumptions and learn from one another. Many of their best ideas emerge from conversations that wander beyond the immediate task at hand. AI excels at producing answers. Organizations excel when people build understanding together. ## **The human cost of efficiency** Efficiency is one of the great promises of AI. It gives faster responses, decisions, and execution. Organizations naturally embrace tools that help people accomplish more in less time. But speed is only one measure of organizational health. The same systems that reduce friction can also reduce opportunities for discussion, context-sharing, and reflection. Those activities often appear inefficient when measured against a task list, yet they help teams learn, adapt, and make better decisions. Organizations succeed by moving information efficiently. They also succeed by creating alignment, developing people, and building shared understanding. Those outcomes rarely appear on a dashboard, but they shape the quality of decisions, the resilience of teams, and the strength of workplace culture. As AI continues to improve the efficiency of work, organizations will increasingly determine what they value beyond efficiency itself. ## **Rewriting the interface** The next time you draft a Slack message or make a request of a colleague, take a moment to consider the language you're using. Are you having a conversation, or issuing a prompt? Before sending a context-free directive, consider adding back some of the elements that AI interactions remove. Explain why the task matters. Share the broader context. Ask a question instead of issuing a command. Take a moment to acknowledge the person on the other side. None of these actions are efficient in the machine sense, and that is precisely the point. We spent decades worrying about whether artificial intelligence would become too much like humans. But the more immediate risk is humans becoming a little too much like machines.

Fable 5 model review: early signals from code review and coding tasks

Juan Pablo Flores — Tue, 09 Jun 2026 00:00:00 GMT

Fable 5 is worth testing for autonomous coding work, especially when the prompt is incomplete and the agent has to discover the environment before it can build. For production code review, the current baseline and Opus 4.8 still look safer. Fable 5 is the kind of model that changes how an agent feels when the task is underspecified. It directs exploration well: first learning the environment, then identifying what files, tools, and constraints are available, then building from that grounded picture. It does not spend much time narrating what it is about to do. If it has enough context, it starts building. We saw that across multiple coding projects we used to test the model's capabilities. We could give Fable 5 vague prompts and still get complete projects rather than prototype shells. It also found solution paths that felt less obvious, including approaches that earlier model reviews struggled to reach without more hand-holding. The same behavior also shows up as a cost. In our coding task benchmark, Fable 5 often kept working until the harness cut it off. That makes the model feel capable, but it also makes it expensive and slower in agent workflows that do not have strong stop rules. So the recommendation is not a clean "switch everything." It is: use Fable 5 where autonomy is the product, keep the current code review path while precision and comment volume are tuned. ### **The decision in one paragraph** Use Fable 5 when the job is to explore, plan, and build, especially when the task can take longer in exchange for a more thorough implementation. Keep the current reviewer in place for now. The code-review signal is close on coverage, but not yet strong enough on precision or volume to become the default. ## **What’s new in Fable 5** Fable 5 is positioned as a Mythos-class model for autonomous knowledge work and coding, which sets a different bar than a routine model upgrade. The promise is not just better answers or a faster Opus 4.8. It is a model built for longer-running agent work: holding more context, making a plan, and carrying the task further before it needs a human to step in. The launch constraints are still important to the capability story. The model includes blocking classifiers for some cybersecurity and biology requests, and the product supports opt-in fallback to Opus 4.8 after classifier blocks. For developers, the practical takeaway is simple: use Fable 5 when the task needs depth, and keep the existing path for workflows that need predictable speed or precision. The public launch price in the release brief is $10 per million input tokens and $50 per million output tokens, with a 10 percent surcharge on regional endpoints. We have seen prices like this before from reasoning models, especially when they introduce a new model category. Those costs may come down over future iterations, but the early picture is clear: developers should evaluate Fable 5 by cost per solved task, not just by token price. ## **Code review: Close on coverage, weaker on precision** In the 105-EP code review benchmark, Fable 5 stayed close to the current baseline on finding coverage. It passed 65 of 105 actionable EPs, just behind the baseline and Opus 4.8 at 66 of 105\. If every comment type is counted, Fable 5 slightly beat the baseline, with 74 of 105 full EP passes versus 72 of 105\. The weaker part is precision. Fable 5 landed at 32.8 percent actionable precision and 19.4 percent full precision, while Opus 4.8 reached 35.5 percent and 26.5 percent. It also produced 253 comments, more than either comparison run, with a large increase in assertive and nitpick-style output. That combination matters for code review because noisy comments create reviewer work even when coverage looks competitive. ![Bar chart shows code review benchmark across three systems and metrics.](/content/assets/image1-16.png) The category breakdown makes the result feel more uneven than the headline coverage number suggests. Fable 5 has useful breadth, but the gains are not consistent across the kinds of issues developers expect a reviewer to catch. In practice, teams should expect helpful findings, while still needing manual triage and fallback coverage for categories where review trust is harder to earn. The harder examples make the rollout case more cautious. On difficulty 4 EPs, Fable 5 passed 8 of 16, behind the baseline at 10 of 16 and Opus 4.8 at 9 of 16\. That does not make Fable 5 a poor reviewer, but it does mean developers should expect more human judgment on the cases where review trust is hardest to earn. ![Table comparing Current baseline, Fable 5, and Opus 4.8 software performance metrics.](/content/assets/image2-10.png) ## **Security: Helpful, but not automatic trust** Developers are likely to experience Fable 5 as more security-aware than a generic coding model, especially when the task asks for careful implementation around risky behavior. That does not mean it should be treated as a drop-in security reviewer. The practical posture is to use it for deeper security-sensitive coding work, then keep the review bar high before trusting the findings. The strongest security signal came from active implementation. In our coding task benchmark, Fable 5 completed a security-relevant Bandit task when it had a clear objective and enough time to work through the code. Fable 5 looks more useful when security is part of a concrete coding task than when it is asked to catch every issue in review. ## **Coding task benchmark: Capable, but long-running** We stopped the coding task benchmark early after a clear pattern emerged: Fable 5 could make meaningful progress, but many tasks ran long enough to hit the agent timeout. That makes this section an experience signal, not a final leaderboard score. The useful story is how the model behaves on real coding work when the task keeps expanding. ![Bar chart details coding task outcomes: 19 timeouts, 6 passed, 4 failed, 4 cancelled.](/content/assets/image3-5.png) The outcome mix should be read as a signal about agent behavior, not as a completed benchmark score. The signal cuts both ways. When Fable 5 finished, it produced serious patches rather than shallow edits. When it struggled, it kept exploring longer than the harness could support. For developers, that makes it a better fit for work where depth is worth the wait, provided the agent has clear limits on time, steps, and tokens. The completed tasks also suggest where Fable 5 is most useful. The wins came from implementation work with real structure: types, APIs, publishing behavior, query logic, caching, and security-relevant code. The misses were useful too. Some were quick wrong turns, while others showed the model investing real effort without converging. That is the tradeoff developers should plan for: more depth, but not always a clean finish. ![Bar charts comparing average agent time and output tokens for passed versus failed coding tasks.](/content/assets/image4-4.png) Fable 5 spends real time and output budget when it finishes. When it times out, the harness can still burn substantial context. The token profile tells the same story as the timing data. Fable 5 does not just cost more because of list price. It can also spend longer thinking, exploring, and generating before it reaches an answer. Cache discounts may reduce the final bill, and prices may come down over future iterations, but teams should still evaluate the model by cost per solved task. For this kind of agent work, timeout rate, cache behavior, and output token usage matter as much as the published token price. ## **Coding projects show the upside** The coding projects gave the clearest view of Fable 5 at its best. In one project, the model did more than assemble a working surface. It organized the implementation into separate layers for state, decision-making, rendering, and controls, then produced a build that passed. The remaining gaps were the kind you would expect from a first complete version: more robust test coverage, safer state handling, and stricter guards around invalid inputs. Another project showed the same pattern in a more interactive setting. Fable 5 built a working real-time application with a stable loop, procedural visuals, stateful interactions, phase changes, canvas effects, multiple app states, and a successful production build. The issues were not basic completion failures. They were the next layer of engineering work: deterministic tests, small-screen polish, and state edge cases. %[https://youtu.be/HZ_d3MGlHDU] This is the clearest qualitative difference from earlier model reviews. With enough context, Fable 5 moves directly into implementation instead of over-explaining the plan or repeatedly asking for permission. It also seems to spend more effort on architecture, interactions, and product shape than a narrower code-completion model would. ## **Recommendation: Use Fable 5 selectively** The recommendation is selective adoption. Fable 5 is worth testing for autonomous coding work, especially tasks that benefit from deeper planning, multi-file execution, and extra time spent on implementation. I would not make it the default for production code review yet. ### **Where I would use it** * Autonomous coding projects with a clear goal. * Multi-file implementation where depth is worth the wait. * Agent workflows with explicit time, step, and token budgets. ### **Where I would hold back** * Default code review traffic before precision tuning. * Security-review positioning without stronger security-specific evidence. * High-throughput workflows where long runs create cost or latency risk. For code review, keep the current baseline or Opus 4.8 path as the default until Fable 5 improves on precision and comment volume. For [coding agents](https://www.coderabbit.ai/guides/types-of-ai-coding-agents), Fable 5 is more compelling, especially when the work benefits from exploration and deeper implementation. The guardrail is operational: give it clear budgets, stop conditions, and review checkpoints. For security workflows, position it as useful for security-sensitive implementation, not as proof of better security review.

CodeRabbit now supports NVIDIA Nemotron 3 Ultra

Sahil Mohan Bansal — Thu, 04 Jun 2026 00:00:00 GMT

*TL;DR: NVIDIA Nemotron 3 Ultra delivers accurate and fast throughput in CodeRabbit's self-hosted AI code reviews*. We are excited to share that CodeRabbit is expanding its support for the NVIDIA Nemotron family of open models, expanding to include Nemotron 3 Super and [**Nemotron 3 Ultra**](https://blogs.nvidia.com/blog/nvidia-gtc-taipei-computex-2026-news/#nemotron-3-ultra) for AI code review workflows. Nemotron 3 Super helps with [context gathering](https://www.coderabbit.ai/guides/code-context) and summarization whereas Nemotron 3 Ultra helps generate code review comments for many reviews outside of the most complex tiers. This expanded support is available for CodeRabbit's self-hosted customers running its container image on their own infrastructure. Initial eval results indicate that **Nemotron 3 Ultra** aligns with our current frontier model ensemble for junior-tier engineering assessments, with similar token efficiency while achieving approximately 2x faster response times. OpenAI and Anthropic models remain the primary engines for producing most of the review comments delivered to your Pull Requests. ### Expanding to Super and Ultra: Faster context gathering and lower latency Previously we had announced our [support of Nemotron 3 Nano and Super](https://www.coderabbit.ai/blog/faster-code-reviews-with-nemotron-3-super), where we reported that a blend of open and frontier models allows us to improve the overall speed of context gathering and PR summarization. This blend of open and frontier models is also more cost efficient by routing different parts of the review workflow to the appropriate model family \- PR Summarization with Nemotron and review comments with frontier LLMs. As with the rest of the Nemotron family, NVIDIA is releasing Ultra as a truly open model, with the weights, training data, and training recipe published alongside it. That openness is part of why Nemotron has been a good fit for self-hosted teams that need to run reviews inside their own environment. With the support for Nemotron 3 Nano, Super and now Ultra, we can use Nemotron open models for context gathering, PR summarization, and some aspects of review comment generation. ![A detailed technical architecture diagram showcasing CodeRabbit's AI agents, context enrichment, and knowledge base components.](/content/assets/code-rabbit-architecture-with-nemotron-1.png) ### What Ultra unlocks beyond Nano and Super When you open a Pull Request (PR), CodeRabbit’s code review workflow is triggered starting with an isolated and secure sandbox environment where CodeRabbit analyzes code from a clone of the repo. In parallel, CodeRabbit pulls in context signals from several sources: - Code and PR index - Linter / Static App Security Tests (SAST) - Multi-repo code graph - Coding agent rules files - Custom review rules and Learnings - Issue details (Plan file, Jira, Linear, GitHub issues) - Public MCP servers - Web search A lot of this context, along with the code diff being analyzed, is used to generate a PR Summary before any review comments are generated. Summarization is at the heart of every code review and is the key to delivering high signal-to-noise in the review comments. We continue to support Nemotron 3 Nano and Super for the repetitive work of context processing during review summarization, which is critical for our code reviews. ![Code editor displaying JavaScript functions and variables, with a related forum discussion below it.](/content/assets/image.png) We compared Nemotron 3 Ultra against equivalent frontier models and our analysis found that at the junior reviewer tier Nemotron 3 Ultra: * Matched our frontier model blend on review quality measured by pass rate and precision * Produced a comparable volume of review comments at a comparable number of tokens per review * Ran with almost 50% less median latency making it 2x faster than using only frontier LLMs These results held for both trivial and junior-level review comments. These are early, encouraging results in a specific place: faster, lower-effort reviews where an efficient open model can carry more of the load. For customers this means faster PR summarization, context gathering and faster code reviews without compromising quality. We are also delighted to support the announcement from NVIDIA today about the expansion of its Nemotron family of open models and are excited to work with the company to help accelerate AI coding adoption across every industry. [Get in touch](https://www.coderabbit.ai/contact-us/sales) with the CodeRabbit team to access CodeRabbit’s container image if you would like to run AI code reviews on your self-hosted infrastructure.

Nemotron 3 Ultra makes the case for fast, open coding models

Juan Pablo Flores — Thu, 04 Jun 2026 00:00:00 GMT

NVIDIA Nemotron 3 Ultra does not feel like another model built primarily for a chat window. The first question is not whether it can win a leaderboard, but whether it can fit into the way developers actually use models now: inside terminals, review pipelines, coding agents, test generators, and workflows where the model has to keep moving through messy context. [NVIDIA](https://www.nvidia.com/en-us/) is releasing a large open model with roughly 550 billion total parameters and about 55 billion active per token, but the real pitch is speed plus control. If a model is fast enough, a developer can stay in the loop. A system can retry it. A coding harness can keep it working until the task is actually finished. Ultra is not the model I would frame as "the new best coding assistant." It points toward a world where open models become fast, controllable workers inside developer systems, not just chat interfaces waiting for the next prompt. For workflows where the model is one part of a larger loop, Nemotron 3 Ultra becomes especially relevant: code review, test generation, repository research, agentic coding, and internal automation where teams care about speed, control, and where the model runs. ![Scatter plot showing AI intelligence index versus output speed, with a most attractive quadrant highlighted.](/content/assets/image-1.png) ## What we know about Nemotron 3 Ultra Nemotron 3 Ultra is the largest model in [NVIDIA's Nemotron 3 family](https://www.coderabbit.ai/blog/coderabbit-ai-code-reviews-now-support-nvidia-nemotron). The family includes Nano, Super, and Ultra, all designed around agentic AI applications. Ultra is the big reasoning engine in that lineup: roughly 550 billion total parameters, with about 55 billion active per token through a sparse mixture-of-experts design. The cleanest comparison is with [Nemotron 3 Super](https://www.coderabbit.ai/blog/faster-code-reviews-with-nemotron-3-super), the previous large model in the family. | Characteristic | Nemotron 3 Super | Nemotron 3 Ultra | | :---- | :---- | :---- | | Role in the family | High-throughput reasoning model for agentic workflows | Largest Nemotron 3 reasoning model for more complex coding, research, and enterprise workflows | | Total parameters | 120B | 550B | | Active parameters | 12B active per token | 55B active per token | | Architecture | Hybrid Mamba-Transformer MoE | Hybrid Mamba-Transformer MoE | | Expert design | Latent MoE | Latent MoE | | Context length | Up to 1M tokens | Up to 1M tokens | | Efficiency features | Multi-token prediction and NVFP4 training/deployment path | Multi-token prediction and NVFP4-oriented deployment path | | Best fit | High-volume agentic workflows, coding, planning, and tool use | More demanding developer workflows where speed, scale, and stronger reasoning need to sit in the same loop | In simpler terms: this is not just a bigger dense Transformer. Ultra is built to activate only part of the network per token, keep long context practical, and produce tokens quickly enough that developers can use it interactively instead of treating it like a slow background batch job. ![Flowchart detailing Nemotron 3 Ultra's fast architecture: long context, Mamba-Transformer, MoE routing, multi-token prediction.](/content/assets/image1-19.png) The launch numbers put Ultra in a strong spot. [Artificial Analysis](https://artificialanalysis.ai/) reported Nemotron 3 Ultra at 48 on its Intelligence Index, making it the leading US open-weight model in that snapshot, ahead of Gemma 4 31B, Nemotron 3 Super, and gpt-oss-120b. Kimi K2.6 still sits higher at 54, so the claim is not that Ultra owns the entire open frontier. The claim is that it is unusually fast for the intelligence level it reaches. Artificial Analysis also reported more than 300 output tokens per second on a pre-release DeepInfra endpoint. For developers, that speed is the useful part. In coding, latency changes behavior. If a model is slow, you fire and forget. If it is fast, you stay in the loop, ask follow-ups, run multiple attempts, and let an agentic harness keep pushing. ![NVIDIA Nemotron 3 Ultra announcement slide showing a cost comparison graph presented by a speaker.](/content/assets/image3-9.png) ## What is different this time Nemotron 3 Super already showed that NVIDIA could build a capable open model for agentic workflows. Ultra pushes further in two ways. First, it is much bigger. Super is around 120B total parameters with roughly 12B active. Ultra moves to roughly 550B total and 55B active. That extra scale shows up in the way NVIDIA and early testers talk about it: not as a small efficient helper model, but as a model that can start taking work from proprietary frontier systems in selected workflows. Second, Ultra appears to have been trained and evaluated with developer harnesses more directly in mind. NVIDIA mentions that Super turned out to be good in agentic harnesses, while Ultra was built with those harnesses in mind. For coding tools, that changes the requirements. A model that works well in OpenCode, OpenHands, Kilo Code, Continue, or an internal code review loop has to do more than answer questions. It has to follow tool protocols, manage long context, make progress under repeated prompts, and recover when it gets stuck. Ultra's behavior fits that target. The model is quick, direct, not especially verbose, and unlikely to ask for lots of clarification. That can be a strength in a harness, but a weakness if the task depends on unstated requirements. It benefits from explicit instruction. The best mental model is closer to Codex-style prompting than Claude-style prompting. Spell out the task. Give acceptance criteria. State the expected output format. ## CodeRabbit Benchmark Performance CodeRabbit's internal benchmark gives a more grounded view than launch charts alone. The benchmark compares a baseline set of review models against a Nemotron 3 Ultra configuration across 105 evaluation problems, ranging from easier issues to harder review tasks. The evaluation uses post-pipeline final comments after verification, deduplication, and assertive filtering. The judge was gpt-5.1 with medium reasoning, low verbosity, single mode, and three votes. ![CodeRabbit benchmark table compares Baseline average and Nematron 3 Ultra performance metrics.](/content/assets/image4-6.png) The top-line result is close: * Baseline average, N=3: 60/105 pass actual, or 57 percent * Nemotron 3 Ultra average, N=2: 58/105 pass actual, or 56 percent * Baseline pass full: 66/105, or 63 percent * Nemotron 3 Ultra pass full: 65/105, or 62 percent * Baseline precision actual: 34.0 percent * Nemotron 3 Ultra precision actual: 33.0 percent The positive read: on this review workload, Ultra was roughly in the same band as the baseline on pass metrics. It found real issues, survived the review pipeline, and produced useful CodeRabbit-style comments. The caveat is reliability. The model had a high retry rate. The benchmark summary shows an average of 36.5 retries for the Ultra runs, compared with 0.3 for the baseline. The retry distribution notes that about 66 percent were scratchpad-only. In practice, the model sometimes voluntarily stops before producing the required output marker or final structured output. Retrying without changing the prompt often works, which suggests the capability is there, but the first-attempt completion behavior is not stable enough to ignore. The practical finding from the CodeRabbit data is clear: Nemotron 3 Ultra can do the work, but it should be wrapped in validation and retry logic for structured-output tasks. There is also an interesting latency signal. In the benchmark, the Ultra run shows a mean latency of 7:06 per full review trace, compared with 8:31 for the baseline. That is not an enormous difference in this specific report, but the Ultra runs were carrying a large retry burden and still remained competitive on time. NVIDIA's framing around Ultra repeatedly returns to the same idea: if the model is fast enough, several attempts can still beat one slower, more careful attempt. The cost story is less clean in the benchmark. The reported total cost for the Ultra run is higher than the baseline in this specific table. That should not be over-generalized, because internal fallback rates, hosted endpoint pricing, and retry behavior can dominate a local experiment. The public NVIDIA and Artificial Analysis story is about cost-to-completion and throughput. The [CodeRabbit](https://www.coderabbit.ai/) results say something narrower: on this benchmark, quality was close, speed was competitive, and the reliability control loop needs work. ## Where Ultra looks strong for developers The strongest use case for Nemotron 3 Ultra is not "replace every coding model." It is "run a lot of useful developer work quickly, with explicit instructions and external checks." It looks promising for: * Code review pipelines where comments can be verified, filtered, deduplicated, and retried * Integration test generation, especially when the model needs to read broad context * Repository research tasks that require scanning many files or documents * Agentic workflows where a harness can keep the model moving until the task is complete * Everyday coding tasks that benefit from fast iteration more than perfect one-shot reasoning NVIDIA also shared a useful example: Ultra was used in OpenCode to read several papers and reason across them. That is not a PhD-level coding challenge, but it is exactly the kind of everyday developer task where speed changes the workflow. You can stay in the terminal, watch the model move, and keep steering. For CodeRabbit-style work, the model also seems especially interesting on easier and medium-difficulty review tasks. These are still valuable reviews: the system needs to catch practical issues, explain them clearly, and produce a lot of review output without waiting on a more expensive frontier model every time. ## What developers should watch out for Ultra needs structure. If you are using it for coding or developer automation, do not treat it as a free-form chat model and hope it infers the workflow. Give it a harness. Give it a checklist. Give it stop conditions. Give it output validation. Practical guidance: * Use explicit prompts with concrete acceptance criteria. * For structured output, validate the required markers or schema before accepting the response. * Add retry logic for premature stops. * Use goal loops or external completion checks so the model keeps working until the task is actually done. * Ask for tests explicitly. In early hands-on use, the model did not always generate its own tests. * Be specific about design requirements. It can produce better visual artifacts than expected, but design is not its core strength. * Prefer it for high-throughput workflows where several attempts are acceptable. * Be cautious for workflows where a single malformed output can break production automation. This model also changes how teams should think about benchmarking. A pure one-shot benchmark may underrate Ultra if the real product loop allows retries. A benchmark that ignores retries may overrate it if the product needs strict first-attempt formatting. The right metric is probably closer to time-to-usable-completion, with quality, retries, latency, and cost all measured together. ## Verdict Nemotron 3 Ultra is one of the most interesting open model releases for developers because it is not only chasing intelligence. It is chasing usable throughput. The model is big, open, and fast. Public benchmarks put it near the top of US open-weight intelligence while keeping it far ahead of many peers on output speed. CodeRabbit's benchmark adds a more sober picture: Ultra can perform close to a strong review baseline, but it currently needs retries and external validation for structured-output reliability. The verdict is nuanced. If you want a model that will nail every strict format on the first try, Ultra is not yet the safest default. If you are building an agentic developer system where the harness can validate, retry, and keep pressure on the model until the work is complete, Ultra becomes much more compelling. For coding teams, the bigger story is not whether Nemotron 3 Ultra replaces a favorite chat model. It is whether open, high-throughput coding agents are starting to feel practical. Try it on [CodeRabbit PR reviews](https://app.coderabbit.ai/login?free-trial=) now and let us know your thoughts.

You’re addicted to AI code generation. Now what?

David Loker — Tue, 02 Jun 2026 00:00:00 GMT

Software engineers are rapidly building a psychological reward loop around tools they only partially trust. It is, indeed, a strange paradox of sorts. Developers distrust these assistants just enough to double-check their work, yet they rely on them enough to keep them permanently open in their IDEs. The immediate challenge for engineering organizations isn't deciding whether to use AI, but rather designing production systems that can handle this unprecedented influx of code without burying teams under a mountain of verification debt. ![Habit loop diagram showing Trigger, Routine, Reward, and central Reward Loop.](/content/assets/internal-reward-loop-image.png) If you want to see how deeply ingrained this has become, try turning off your coding assistant for a single afternoon. Without AI, development reverts to its traditional, deliberate pace. You start by digging through documentation, manually tracing unfamiliar modules, and facing a blank file trying to figure out where to begin. You might copy a design pattern from a neighboring microservice, slowly writing the first iteration line by line while trying to hold the broader architecture, edge cases, and business logic entirely in your head. With an AI assistant, your entire entry point shifts. Instead of staring at a blank screen, you ask the model to explain the module, outline the likely call path, and offer perhaps three different implementation approaches. You have it scaffold the basic service, draft the database migration, and sketch out the initial test cases. While you still have to evaluate and correct the output, you are starting the day reacting to an existing draft rather than building from scratch. This shift in momentum is exactly what makes working without an assistant feel so jarring. The core responsibilities of the job haven't changed, but the tempo clearly has. Without the AI-coding tool, you lose the instant second opinion, the built-in explainer, and the immediate gratification of seeing a prompt materialize into a functional boilerplate. The work is entirely doable, but it feels significantly heavier once your daily workflow has adapted to that accelerated feedback loop. This structural shift in the [software development lifecycle](https://www.coderabbit.ai/guides/agentic-sdlc) is an addiction or even stronger than an addiction. The genie is out of the bottle, and there’s no going back. Let’s face it, software engineers have developed a functional dependency on a tool that provides immediate feedback. You prompt, edit, accept, and run the code in a continuous cycle. Sometimes the solution is incredibly elegant. Sometimes it is entirely wrong but highly confident. Often, it is just close enough to keep you moving forward. Ultimately, AI tools have fundamentally changed the day-to-day experience of programming. ## From tech novelty to muscle memory AI code generation has quickly transitioned from a tech novelty to muscle memory. According to [Stack Overflow’s 2025 Developer Survey,](https://survey.stackoverflow.co/2025/) 84% of respondents were using or planning to adopt AI tools, with over half of professional developers utilizing them daily. [JetBrains reported](https://blog.jetbrains.com/research/2025/10/state-of-developer-ecosystem-2025) similar findings, noting that 85% of developers regularly use AI tools in their development process, and 62% rely on a dedicated AI coding assistant or editor. At major tech companies, the adoption curve is steep. In early 2026, Google CEO Sundar Pichai [noted](https://blog.google/innovation-and-ai/infrastructure-and-cloud/google-cloud/cloud-next-2026-sundar-pichai) that 75% of all new code at the company was AI-generated and subsequently reviewed and approved by engineers. That’s a significant jump from 50% just six months prior. He highlighted a complex code migration that was completed six times faster through human-agent collaboration than would have been possible using traditional methods. The key phrase here is *AI-generated and approved by engineers*. The shift is here to stay, and the real opportunity lies in building robust review systems that help teams translate this engineering speed into secure, stable production code. Historically, writing code was expensive and time-consuming. Engineering organizations built their cultures around that scarcity through structured sprint planning, estimates, peer reviews, and strict release trains. Today, drafting code is cheap. But trusting it remains exceptionally resource-intensive. This is the new economic reality of the software engineering addiction. To be sure, code generation is abundant, but verification is scarce. ## Addiction rhymes with contradiction The most glaring contradiction in modern software development is that engineers are using AI more while trusting it less. The Stack Overflow survey also found that 46% of developers actively distrust the accuracy of AI tools compared to only 33% who trust them. Unsurprisingly, senior engineers were the most skeptical demographic. Paradoxically, this behavior sounds entirely contradictory on paper, but it makes complete sense as a practical workflow decision. AI assistants reduce the cognitive friction of getting started. They turn a daunting, empty file into an editable draft, convert an unfamiliar codebase into a conversational QA session, and quickly build out repetitive test suites. Traditional, or manual-first software development is plagued by tiny, annoying frictions like hunting down a forgotten method signature, navigating poorly documented internal conventions, or writing rare migration syntax. AI smooths over these roadblocks just enough to keep the developer moving forward. Behavioral research shows that variable rewards and rapid uncertainty resolution can make digital experiences highly compelling. AI coding assistants mirror this pattern perfectly. One prompt generates standard boilerplate, the next hallucinated an internal library, the third hits the exact solution, and the fourth delivers a broken implementation wrapped in flawless formatting. AI acts less like outsourcing and more like a tool that sustains your personal creative momentum. ## Productivity metrics meet reality The productivity gains are real, but they come with strings attached. [DORA’s 2025 research](https://dora.dev/research/2025/dora-report) indicated that 90% of tech professionals use AI at work, with over 80% reporting a noticeable boost in productivity. AI excels at explaining complex logic, translating languages, generating initial test coverage, and minimizing the drag of routine tasks. It helps junior developers self-start and frees up senior engineers from tedious boilerplate. However, productivity metrics in software engineering have always been difficult to isolate. A single developer can feel incredibly fast while the broader organization slows down. A team might close more tickets and push more code, only to dramatically increase their code review burden and quietly introduce architectural flaws, security vulnerabilities, and operational surprises down the line. A 2025 [randomized controlled trial by METR](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) illustrated this tension clearly. In a study involving experienced open-source developers working inside familiar repositories, engineers actually took 19% *longer* to complete tasks when allowed to use AI tools. While METR’s [2026 update](https://metr.org/blog/2026-02-24-uplift-update/) noted that newer models have likely improved this speed metric, it also emphasized that measuring true productivity has become much more complex as adoption spreads. DORA’s 2026 [updated analysis](https://dora.dev/insights/balancing-ai-tensions), meanwhile, offers an accurate framework for this shift. AI accelerates the initial creation of code, but the time saved during the drafting phase is simply reallocated to auditing, testing, and verification downstream. Consequently, higher AI adoption has been statistically linked to both an increase in delivery throughput and an increase in software deployment instability. Think of AI as a multiplier for whatever environment it drops into. If a team already has robust automated guardrails through strict linting, heavy test coverage, and comprehensive integration environments, they can safely handle this influx of code. But if an organization is already struggling with flaky tests, fuzzy code ownership, and slow review cycles, these tools just help them pile up [technical debt](https://www.coderabbit.ai/blog/vibe-coding-because-who-doesnt-love-surprise-technical-debt) at a terrifying new speed. The old workflow used to be about writing, reviewing, and merging. The new reality is about generating, verifying, and taking actual accountability for the output. That is the operating model behind [guardrails for agentic coding workflows](https://www.coderabbit.ai/guides/guardrails-for-agentic-coding-workflows). ## From raw code production to system stewardship AI-generated code looks highly professional on the surface. It uses standard naming conventions, includes helpful comments, and mimics existing codebase patterns. This polished appearance can easily distort the psychology of a code review. While messy code practically begs to be challenged, clean-looking code encourages quick approval. The most dangerous architectural failures such as missed multi-tenant separation, weak permission boundaries, or subtle race conditions hide easily behind this visual confidence. Data from CodeRabbit’s [2025 analysis of open-source pull requests](https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report) suggested that AI-assisted changes averaged roughly 10.83 issues per PR, compared to 6.45 for entirely human-authored code. Third-party analysis by *The Register* [noted](https://www.theregister.com/software/2025/12/17/ai-authored-code-needs-more-attention-contains-worse-bugs/) that this shouldn't be a reason to abandon AI tools altogether, but rather an indicator of a shifting review burden. AI introduces errors at a different scale and volume than traditional coding. Because of this, the role of the developer is shifting away from raw code production toward system stewardship. When drafting code becomes trivial, human value centers on intent, architecture, risk management, and ultimate ownership. Rather than focus on syntax, senior engineers must learn to dictate explicit constraints when it comes to security assumptions, performance expectations, data boundaries, and more. A thorough, AI-assisted engineer has to ask targeted system questions: * What architectural invariants must this code preserve? * Where could this change introduce regressions across different user roles or regions? * What specific test would fail if the model misunderstood our internal business logic? * Does this proposal duplicate an existing internal abstraction? ## Channeling the prompter’s high To adapt to this environment, engineering teams need to modernize their code review rituals rather than restrict tool usage. The goal is to make AI integration visible, teachable, and strictly aligned with production standards. This requires three core principles: * **Generate openly:** Teams should feel free to use AI as a collaborator for drafting, debugging, and documentation. The organization should focus on whether the final change is fully understood by the author, not whether an AI touched the keyboard. * **Verify aggressively:** Automated testing, security scanning, and observability must keep pace with generation speed. The faster a draft appears, the faster the feedback loop needs to react. * **Own completely:** Blaming an AI for a production outage carries the same weight as blaming a compiler. It describes the mechanism of failure but absolves no one of responsibility. In practice, feeding this AI tool dependency means teams have to completely change how they police code. Since the addiction to instant drafting inevitably floods repositories with massive diffs, teams must learn to prioritize reviewing systemic risk over cosmetic code style. Survival requires paying ruthless attention to data access, concurrency, and security boundaries while consciously protecting deep focus time from the compulsion of constant prompting. Ultimately, the goal is to design an engineering system that channels this craving for momentum into deliberate judgment. AI tools excel at feeding that developer need for immediate progress and in turning friction into instant motion. The teams that thrive won't try to cure the addiction. Instead, they will graduate from treating it as a private productivity fix and start managing it as a core, high-volume workflow.

Opus 4.8 benchmark results for AI code review and code generation

Juan Pablo Flores — Thu, 28 May 2026 00:00:00 GMT

Anthropic just shipped Opus 4.8. Before its release, we spent some time putting it through its paces, most of it on code review tasks. We ran it against our standard evaluation harness, watched how it behaves on real pull requests, and probed where it holds up and where it strains. Alongside that, we used it for the kind of long-running coding work that tends to break agents before they finish. On review, it lands at parity with some of our tuned production ensemble. The surprise was how much it pulled ahead on code generation and long-horizon agentic sessions. ## **What’s new in Opus 4.8** Three things actually shipped, and everything else is downstream: - ** Long-horizon agentic execution.** Performed well in tasks that span many tool calls without losing the thread. It plans before acting and holds the goal across hours-long sessions. Give it the full spec up front at high effort. Drip-feeding requirements perform noticeably worse. It completed more multi-hour, many-file sessions without dropping the thread than any model we've evaluated, and the same intermediate reasoning shows up in stronger code generation. - **Mid-session system prompts.** The messages array now accepts `{"role": "system", ...}` entries mid-conversation without invalidating prompt caches. The model follows them most reliably as context rather than overrides. It also narrates its plan, second-guesses, and requests more permission than prior Opus versions, all useful behaviors, but ones that require active budgeting and steering. - **Tool-use recalibration.** Web search triggers more often, but runs fewer rounds. Retrieval tools, sub-agents, and memory files trigger less often, defaulting to answering from context. The net effect is high-precision, low-recall behavior, steerable with an explicit instruction. On code review, it lands at parity, with an actionable pass rate of 61% vs 62%, and full-system 72% vs 68% at unchanged precision. But the comment mix shifts and critical findings dipped (35 to 29), which gives us pause. In our “Results” section below, we dig into why, and whether it is recoverable. CodeRabbit is integrating it selectively where its strengths fit, and routing other models that win on cost without sacrificing quality or pass rate. %[https://youtu.be/LzgPzQud0zA] ## **What we tested** We ran Opus 4.8 through the same harness we use for every model release: 100 open-source pull requests sampled across trivial, minor, and major complexity tiers. We compared two thinking configurations (a default escalating medium/high/x-high by tier, and a lower-thinking variant running low/medium/high) against a baseline running our current production model mix on the same PRs. Two metrics drive the analysis: pass rate (the fraction of PRs on which the model surfaced the equivalent of what a senior human reviewer would flag) and precision (the fraction of comments that were actionable rather than noise). "Actionable" is adjudicated by senior reviewers. ## **Results** The default Opus 4.8 config edges past our baseline on full-system pass rate (+4pp, 72% vs 68%) and sits within noise on actionable pass rate (61% vs 62%). Precision holds at 33.8% on actionable comments and ticks up a point on the full system. For a model going head to head with a tuned ensemble on a surface it was not specifically optimized for, that is a strong result, and the cross-file reasoning is clearest on senior-tier PRs. ![Severity distribution bar chart compares Baseline, Opus 4.8 default, and Opus 4.8 One Thinking models.](/content/assets/opus-4-8-image1.png) ![Data table displaying findings by severity for Baseline and Opus 4.8 models.](/content/assets/opus-4-8-image2.png) The comment mix is noisier than baseline, however. Major findings drop from 119 to 81, while minor and nitpick findings both roughly double. The model is shifting volume from the middle of the severity range toward the bottom. The one result that gives us pause is critical findings, which fell from 35 to 29. For a code-review tool, missed criticals matter more than any other category of finding. Our working explanation is that Opus 4.8 follows review instructions literally. Consequently, conservative prompts ("only report high-severity issues") suppress recall more than they did with prior models, and that the higher-severity bug-finding capacity is real once the model is allowed to report broadly and we filter downstream rather than constraining at the source. The lower-thinking variant tells a useful secondary story. Cutting reasoning effort drops precision four points and actionable pass rate five points. Thinking level is a first-class configuration decision. We also found the default config costs more. We measured $0.20 to $0.28 per call against roughly $0.13 for [Opus 4.5](https://www.coderabbit.ai/blog/opus-45-for-code-related-tasks-performs-like-the-systems-architect) and $0.04 to $0.12 for [Sonnet 4.5](https://www.coderabbit.ai/blog/claude-sonnet-45-better-performance-but-a-paradox). On code review alone, the model is at parity, making the premium hard to justify for review-only use. What earns its value is on long-horizon agentic and code-generation work below. That cost-versus-surface tradeoff is exactly why we route it selectively rather than everywhere. ## **Where it struggled** Performance degrades visibly once context crosses 200k tokens. The model slows and starts to miss references and edge cases it would have caught cleanly at lower context windows. This is an observational finding from hands-on use, not a controlled measurement. CodeRabbit's context engine works around this, but teams using Opus 4.8 directly will hit a wall in monorepos and large codebases. ## **What this means for CodeRabbit users** We are integrating Opus 4.8 selectively. Its strengths (cross-file reasoning, long-horizon agentic quality, planning under a single up-front spec) show up most on senior-tier changes. So that is where you will see it engaged. For trivial and junior-tier PRs, we continue routing to the models that win on cost and pass rate at those tiers. For our agentic features, we expect Opus 4.8 to be the strongest backbone we have integrated. If you run Opus 4.8 directly, most existing Opus prompts work without modification. A few tune-ups produced measurable differences in our testing. Start at "high" thinking rather than "x-high" and test across tiers. Front-load the full task context for long-horizon work. Add an explicit search-first or delegation instruction to recover depth on research-heavy work. Drop conservative language from review prompts and filter downstream instead. Name the small decisions the model can make on its own. We will keep evaluating as the model and our harness change. If the picture shifts, we will publish updated numbers.

CodeRabbit is now in the Claude Marketplace

Aravind Putrevu — Wed, 27 May 2026 00:00:00 GMT

**Starting today, Anthropic customers can apply their existing Anthropic spend commitment toward CodeRabbit.** Simplified procurement. Consolidated AI spend. A single Anthropic invoice for both. Claude is rewriting how software gets built, and it's faster than anyone thought possible. The question now isn’t: _How fast can we write code?_ Instead it’s: _How much do we trust what’s shipping?_ CodeRabbit is the independent quality gate for the agentic SDLC. Agent-agnostic by design, it catches real bugs, explains what's changed and why, and enforces the standards each team actually cares about. More than 15,000 teams trust CodeRabbit to ship agentic code with confidence. > CodeRabbit is the only review tool I trust when running fully autonomous coding loops. * Abhi Aiyer, CTO, Mastra #### Get CodeRabbit through the Claude Marketplace **For Enterprises:** [Contact CodeRabbit](https://www.coderabbit.ai/contact-us) or your Anthropic account team to add CodeRabbit through your [Claude Marketplace](https://claude.com/platform/marketplace) agreement. **For Developers:** [Install CodeRabbit free](https://coderabbit.link/88vkRqF) on any GitHub, GitLab, Bitbucket, or Azure DevOps repo. For more on how CodeRabbit fits into agent workflows, see [What is the agentic SDLC?](https://www.coderabbit.ai/guides/agentic-sdlc) and [How CodeRabbit built a planning layer on Claude](https://www.coderabbit.ai/blog/how-coderabbit-built-a-planning-layer-on-claude). **Claude writes the code. CodeRabbit makes sure it’s right. The engine of trust for your agentic SDLC.**

AI can migrate your entire codebase. Reviewing it is another story

Santosh Yadav — Tue, 26 May 2026 00:00:00 GMT

Three years ago, I was migrating code that generated a very large diff. My principal engineer was concerned and asked “could we break this PR down into smaller PRs so it would be easier to review.” I explained to them that it was a migration PR, which is why it was much larger than any other PR i’d otherwise submit. ![Values +679 (green) and -34,644 (red) with a five-segment red progress bar.](/content/assets/image1.png) Migration PRs are common, and companies take them on for many reasons, whether it’s moving from React to Angular or modernizing an aging stack. Before AI, though, these efforts were incredibly time-intensive and typically resulted in massive diffs that were difficult to review. Beyond the migration work itself, one of the biggest challenges organizations face is figuring out how to review these PRs effectively. My own migration experience happened before today’s wave of agentic coding tools and advanced models. Back then, generating the migration was the hard part. Today, AI has dramatically lowered that barrier, and the bottleneck is shifting toward something else entirely: reviewing the large, complex diffs that migrations inevitably produce. That same shift is why [AI pair programming](https://www.coderabbit.ai/guides/ai-pair-programming) turns review into the constraint when agents write more of the code. ## Did Bun being rewritten in Rust change migrations? One of the biggest and most recent examples of a migration the world saw recently was Bun being rewritten from Zig to Rust. It involved over 1 million lines of code being modified across roughly 2,000 files, over a few days. Yes, you read that right, a few days. ![GitHub pull request for rewriting Bun in Rust, highlighting project details and benefits discussed.](/content/assets/image3.jpg) In the case of Bun, the project started small and picked a language that was easy to understand and scale. Recently, the project was difficult to scale and dealt with memory leak issues, which was one of the many reasons why Bun migrated off Zig to Rust. Thousands of people were surprised, and even some were angry, as the migration itself was written by AI. I understand the frustration of users, but, it opened up for discussion that migrations at this scale will be more common in the future. ## PR review views were created with the idea that PRs should be small AI coding agents can program in any language, which changes the whole conversation around migrations. Before, teams used to pick a language or framework based on what they were comfortable writing code in, and even hired individuals specifically to the tools they were going to or already use. Slowly, syntax is becoming irrelevant, and the principles and basics of programming as a whole are far more important than ever before. Uncle Bob Martin, famous for his book Clean code, recently wrote on X (formerly Twitter): ![Uncle Bob Martin's tweet emphasizes Clean Code is about structure, not syntax, applying universally.](/content/assets/image2.jpg) [https://x.com/unclebobmartin/status/2055016815047164304](https://x.com/unclebobmartin/status/2055016815047164304) While AI agents are language agnostic for programming, the review process of PRs is still a challenge for organizations or OSS projects, as it’s hard to understand what changed especially in a language you are not familiar with. Sure syntax still matters to an extent, but understanding what is being shipped and discussing it during the PR review is still where most developers struggle, and as PRs are only getting larger, this problem is compounding over time. For years, the PR review view was meant for PRs that are small, so developers could understand what changed during the review process. ## It’s time for a change As a developer, it’s hard to imagine a world before GitHub launched. I still remember sending patches over email to my colleagues, and when GitHub launched and allowed us to see the diff in the browser it changed our code shipping experience as not only an individual but as a team. We all know that with AI writing most of the code nowadays, the size of PRs is only increasing. Despite this new reality, [the diff has been the default review interface](https://www.coderabbit.ai/blog/introducing-change-stack-the-first-ai-native-code-review-interface) for nearly two decades. Take the Bun PR everyone is talking about; the PR has more than 1,300 discussions, and all of them are hidden. All useful discussions in the thread are not visible until you expand them. ![Dark-themed code review screenshot displaying feature upgrade, optimization notes, and a user's critical comment.](/content/assets/image4.jpg) Sure a rewrite of this size is impressive and opens up the door to the possibilities that AI can complete in terms of migrations, but the harder problem to solve is getting reviewers to understand the migrated code at large scale across any programming language. We need a view with details that can explain the intent behind those changes. We are solving this issue at CodeRabbit and I couldn’t be more excited about it. ## What we built: CodeRabbit Review Be honest with yourself for a second. How often do you read the code when doing a PR review? It’s fairly common for reviewers to not have the bandwidth to read through large PRs line-by-line and understand what has changed. I have been writing and reviewing code for close to two decades now and when I review code I look for intent and ensure the code solves the correct problem. Most of us do the same; we check why the PR is open and what problem it's trying to solve. In some cases, you have more context about how the changes can affect the codebase, so you look at the CI/CD pipeline and approve or leave a comment. So, if you think about the PR review process, the two most important things are intent and context. At CodeRabbit, we ship improvements to our context engineering approach daily and have SMEs fully dedicated to this process. With CodeRabbit Review, we are providing developers with everything they need to review a PR with the intent and context behind the changes. ![Dark themed code editor showing source code and a comments pane with review discussions.](/content/assets/image5.jpg) The context and intent regarding the code changes stay where the code is, avoiding the back and forth that comes with context switching when reviewing code. CodeRabbit Review knows which changes should be reviewed together and stacks the changes, which makes the work easier for anyone looking at the PR and understand the changes with better context. ![A list of programming topics including Rust build, migration automation, and core primitives.](/content/assets/image6.jpg) It’s quite common for PR comments to get lost in the discussion and CodeRabbit Review helps navigate these situations and puts the comments right next to your files so they are easy to find to speed up the review process. ## Code review in OSS The open-source world is moving faster thanks to AI, and OSS maintainers can ship bug fixes and validate changes faster than ever. However, in conversations we’ve had with users, discussing large PRs with users is still an issue. CodeRabbit Review can handle this use case and allow maintainers to better understand changes to larger PRs in a faster amount of time. Try CodeRabbit Review on the next large PR in your queue. CodeRabbit Review is free for a limited time for every CodeRabbit user. You can find it by clicking Review Change Stack in the CodeRabbit PR summary comment.

The diff says 1,400 lines. The change is six

Konrad Sopala — Thu, 21 May 2026 00:00:00 GMT

You open the PR, and find it's an AI-authored change, and the diff shows 1,400 lines of code changed. Every moved line shows up as a delete on the left and an identical add on the right. The one line that actually changed is buried somewhere in the middle of all that, indistinguishable from the noise around it. So you do what you always do, and scroll through the changes and mentally subtract the moves until you find the substance. During your review process you are not reviewing what changed, but actually reverse-engineering it and then reviewing whatever’s left of your attention. That's what semantic diff view fixes. ## Meet the semantic diff view in CodeRabbit Review CodeRabbit Review now has a third way to look at a diff. Alongside Unified and Split views, there's a Semantic view that groups related code movement and token-level changes together, so large [pull requests](https://www.coderabbit.ai/guides/agentic-code-review) are actually inspectable (including AI-authored ones). ![A dark themed code editor displays a diff view with removed and added code lines.](/content/assets/image1-8.png) The premise is the same one [Change Stack was built on](https://www.coderabbit.ai/blog/introducing-change-stack-the-first-ai-native-code-review-interface). AI-assisted development made PRs bigger and more frequent. A single change now routinely touches dozens of files across multiple layers. A flat, line-by-line diff was already a rough way to read that. When most of the diff is mechanical movement, it's worse than rough: it's actively hiding the change inside its own bookkeeping. Semantic mode reads the diff the way you would if you had the time to do it by hand. A block that moved is shown as a block that moved, not as a paired deletion and insertion you have to recognize and discount yourself. A token-level edit inside an otherwise-unchanged line is surfaced as the edit, not the whole line lighting up. What's left, once the noise is grouped and labeled, is the part you opened the PR to evaluate. %[https://youtu.be/pYWOBPhVdRU] It slots into everything CodeRabbit Review already does. You're still inside ordered cohorts and layers, anchored to specific line ranges, reading the range summary in the right panel and dropping inline comments that post back to GitHub natively. Semantic is just the lens you put over the diff once you're there. Toggle it from the diff header like you would unified or split. ## What else shipped alongside it [A few other updates](https://docs.coderabbit.ai/changelog#change-stack-diff-views) in CodeRabbit Review worth knowing about: * **All files view:** The left rail now has an All files entry for searching and jumping across every file in CodeRabbit's review scope, not just navigating cohort by cohort. When you already know the file you want, you go straight to it. * **Comment autocomplete:** The comment composer now does GitHub-style autocomplete: user and team mentions, issue and pull request references, metadata references, and emoji shortcodes. The comment you leave in CodeRabbit Review reads the same as one you'd type in GitHub, because it's the same syntax. * **Automatic line wrapping everywhere:** Diff lines now wrap automatically in every view, so there's no separate line-wrap toggle to find and flip. One less control between you and the code. None of these are headline features on their own. Together they remove the small frictions that add up when you're an hour into a large review. ## A quick refresher, if you missed CodeRabbit Review If this is the first you're hearing of CodeRabbit Review: it's a code review interface that reorganizes a pull request from a flat, alphabetical file list into a guided, layer-by-layer walkthrough. CodeRabbit groups related work into cohorts, orders the changes inside each one into layers so foundational changes come before the code that depends on them, writes a plain-language summary for each range, and generates a diagram inline where one actually earns its place. You open it from the **Review Change Stack** button in the CodeRabbit PR comment. It's an opt-in per reviewer. Teammates who want the default GitHub experience just ignore the button, and nothing about the PR changes for them. Full details are in the [documentation](https://docs.coderabbit.ai/pr-reviews/change-stack). ## Try it out The semantic diff view, the all files view, and comment autocomplete are live in CodeRabbit Review now. CodeRabbit Review is on GitHub, in early access, and available to all users currently. It'll be part of the Pro+ plan going forward. Next time an AI-authored PR lands and the diff looks like 1,400 lines of static, open it in CodeRabbit Review and switch to semantic. [Get started with CodeRabbit](https://coderabbit.link/Q5oy5FI)

Explainable reviews: CodeRabbit Review and the context engine that make it possible

Yiwen Xu — Tue, 19 May 2026 00:00:00 GMT

CodeRabbit is the verification layer and quality gate that developers and organizations trust. But a great review does more than point out bugs or suggest fixes. It helps teams trace a change from implementation back to intent: what changed, why it matters, how to improve it, and whether it is safe to ship. That job is becoming harder as developers rely more on coding agents. AI is already changing the shape of software delivery: teams with high AI adoption are merging 98% more pull requests while spending 91% more time on review, according to a [Faros AI study](https://agentmarketcap.ai/blog/2026/04/13/91-percent-pr-review-bottleneck-ai-coding-agents-code-review-crisis-2026) of more than 10,000 developers. When more code is being produced and shipped faster, review cannot just be about catching bugs. Teams need a better way to understand what changed, why it matters, and how it connects to the rest of the system before it ships. This is the idea behind [explainability](https://www.coderabbit.ai/blog/do-you-trust-your-ai-agent). [Harjot Gill](https://www.linkedin.com/in/harjotsgill), our CEO, has framed this as the agentic-era equivalent of cloud observability. As more coding work shifts to AI agents, humans need a new interface for understanding and trusting the output. Explainability is becoming the new observability: the layer that helps teams see, understand and verify what their systems and their agents are doing. ## From understanding the diff to understanding the change Recently we launched [Change Stack](https://www.coderabbit.ai/blog/introducing-change-stack-the-first-ai-native-code-review-interface), a new AI-native review interface that walks reviewers through a pull request like a senior engineer would. CodeRabbit already helps teams understand changes through summaries, walkthroughs, logic diagrams, and actionable review comments. Change Stack builds on that foundation by showing the path through a change, not just what changed, but how the pieces connect. ![GitHub bot comment displaying a "Review Change Stack" button and failed pre-merge checks.](/content/assets/image2.png) As developers write less code by hand and rely more on agents to generate it, review shifts from "is this diff correct?" to "did the system build what we intended?" We call this [reviewing intent](https://www.coderabbit.ai/blog/nobody-is-going-to-read-the-code): verifying the plan and the outcome, not just the keystrokes. The diff has been the default review interface for nearly two decades. CodeRabbit Review shows what becomes possible when the system can explain the shape of a change and help developers verify that it matches their intent. For example, CodeRabbit reviews PRs using semantic diffs. Traditional line-by-line diffs treat a renamed variable, a reformatted block, and a real logic change as equally important, forcing reviewers to separate signal from noise by hand. Semantic diff understands the structure of the change. It filters out irrelevant changes, detects moved code, and surfaces only the changes that matter. Reviewers see what actually changed, so they can review code faster and more accurately. That experience is only possible because of the deeper system underneath: [context engineering](https://www.coderabbit.ai/guides/what-is-context-engineering) that understands intent, connects changes across files, reasons about impact, and explains the path through a pull request. ## What It actually takes to deliver Explainable Reviews: context engine and harness engineering At CodeRabbit, a trustworthy and explainable review starts long before the review comment is written. ### Inside the context engine For every PR, we build the context the model needs to reason accurately. We clone the repo, analyze the diff, and construct a code graph that traces how the change connects to the rest of the codebase cross-file and cross-repo. The context engine then pulls in the surrounding engineering knowledge linked issues, architecture standards, custom review instructions, coding conventions, past PRs, team-specific [learnings](https://docs.coderabbit.ai/knowledge-base/learnings), and through [MCP](https://docs.coderabbit.ai/knowledge-base/mcp-context#how-coderabbit-uses-mcp-during-analysis), the documents and systems your team builds against and filters all of it down to what is relevant to this specific change. Signals from [40+ linters and SAST tools](https://docs.coderabbit.ai/tools) feed in alongside, real-time web queries pull current release notes and newly disclosed vulnerabilities, and everything runs in an ephemeral, isolated sandbox with zero data retention. This is the work behind the work. In our system, we do a lot of work in context enrichment. The hard part isn't asking an LLM to review a diff. It's giving the model the right context, in the right order, at the right level of detail so it can understand intent, reason through impact, and explain its findings clearly. ![Flowchart illustrating an AI-powered code review process emphasizing context enrichment and agent-based verification.](/content/assets/image1-15.png) ### The harness around it The harness around the context engine matters just as much. It routes each task to the right model by complexity: cheap and fast where deep reasoning isn't needed, frontier where it is, open models where they meet our quality bar. A review agent surfaces possible comments; verification agents filter them against the code guidelines, configuration, and team preferences before review comments reach developers. Underneath all of it sits an evaluation framework that tests every model release, prompt change, and context strategy against recall, precision, latency, and cost. That feedback loop is how we improve quality without just throwing more tokens at the problem. This is the system around the system: context retrieval, ranking, filtering, sandboxing, tool orchestration, prompt design, model routing, verification, and evaluation loops all working together. CodeRabbit has spent three years refining this harness across millions of pull requests and more than 15,000 engineering teams. That accumulated domain expertise, knowing which context matters for which kind of change, is the difference between a system that summarizes diffs and one that finds the issues that could derail what you intended to ship. ## Try CodeRabbit Review AI can write the code, but your team is still responsible for what ships. CodeRabbit is the AI-native quality gate that helps teams move fast without losing control. It provides instant explainability for every change and enforces consistent standards across every pull request, so what ships matches what you intended. Try [Change Stack](https://www.coderabbit.ai/blog/introducing-change-stack-the-first-ai-native-code-review-interface) on the next large PR in your queue. Change Stack is free for a limited time for every CodeRabbit user. You can find it by clicking **Review Change Stack** in the CodeRabbit PR summary comment.

A major walkthrough upgrade: explainable PRs and smarter reviewer routing

Konrad Sopala — Thu, 14 May 2026 00:00:00 GMT

Two updates to the CodeRabbit PR walkthrough went live recently, and together they're a meaningful upgrade to what the walkthrough actually does. One changes how you read a PR, and the other changes how it gets routed to reviewers. They sit in different parts of the flow, but they pull in the same direction: the walkthrough should explain the PR, not just describe it. A flat diff describes. A flat list of suggested reviewers describes. Neither helps you actually trust the change, or get it in front of the right people. These two updates do. ## Layer Based Walkthroughs: Read the PR in the order it was actually built You open a PR. The file list is alphabetical. ``` api/ db/ services/ tests/ utils/ web/ ``` You start scrolling and try to reconstruct what happened: was the migration the starting point, or did it come after the API change? Is this new service the cause of the type changes downstream, or a consequence? Where do the tests fit? This is the price of a flat diff. The order you read the changes in has no relationship to the order they were made in, or to which parts depend on which. You spend the first few minutes of every PR sorting that out before you can evaluate anything. It gets worse with AI-authored PRs. They tend to be larger and touch more files at once, so the mental reconstruction tax gets paid more often. And the cost of getting it wrong is higher, because there isn't a human across the desk to ask what they were thinking. We've written before about [explainability](https://www.coderabbit.ai/blog/do-you-trust-your-ai-agent), the idea that AI systems doing real work on things that matter have to show their homework, or the trust never quite materializes. A flat diff is the opposite of that. It tells you what changed, not how the change was reasoned about, what depends on what, or in what order things had to happen for the whole thing to make sense. The burden of figuring that out lands on whoever opens the PR. %[https://youtu.be/nT_YZuaE4Nw] Layer Based Walkthroughs are CodeRabbit's answer. Instead of an alphabetical file list, CodeRabbit reverse engineers how the change was actually built and organizes it into logical layers: what was changed first, what came next, and what depends on what. A typical PR walkthrough might be ordered: 1. Data models and schema changes 2. Backend logic that uses the new model 3. API responses that surface the new shape 4. Frontend or consumers that read the new responses 5. Tests covering the above ![GitHub walkthrough page detailing payment intent and calendar export system changes in a summary table.](/content/assets/image1-14.png) Each layer comes with its own summary scoped to the changes inside it. You can read top-down for the full story, or jump to the layer that matters most to the kind of review you're doing. Security folks tend to want the boundary stuff, frontend folks tend to want the responses. It works the same way whether a human or an AI agent wrote the changes. But the AI case is where it pays off most: when you can't ask the author what they were thinking, the walkthrough has to show the work instead. Layer Based Walkthroughs are how CodeRabbit does that. ## Suggested reviewers instructions: CodeRabbit says who reviews what CodeRabbit has had suggested reviewers in the walkthrough for a while now. The suggestions come from git history and code ownership, which works well in steady-state codebases where the right reviewer for a file is whoever's been touching it recently. It works less well in two cases: * The right reviewer isn't the most recent contributor. They're the subject matter expert for that area: the security team for anything in auth/, the data platform team for any migration, the SRE on call this week for infra changes. * The right reviewer is a team, not a person. [Suggested reviewers instructions](https://docs.coderabbit.ai/pr-reviews/walkthroughs#suggested-reviewers) let you spell this out directly in the CodeRabbit YAML file. Map reviewers, individual users or teams, to the conditions where they should be assigned: ![Suggested reviewer instructions describing mapping users to PR scenarios and GitHub team support.](/content/assets/image2-9.png) ![Code snippet showing review instructions for different developer groups based on pull request changes.](/content/assets/carbon-2.png) When the list is empty, CodeRabbit falls back to its existing prior-PR-based suggestions, so you can be as exhaustive or as selective as makes sense for your team. A couple of small things worth flagging: * Team handles are supported on GitHub. On GitLab, group handles are ignored; only individual user handles get resolved and assigned. * Pair this with auto\_assign\_reviewers: true if you want CodeRabbit to actually request reviews from the suggested folks, not just list them in the walkthrough. ## Try them out If you've been mentally reconstructing the order of changes in every PR you open, Layered Walkthroughs should take a chunk of that off your plate. If you've been writing the same Slack messages over and over, "Hey, can someone from the platform team take a look at this?", suggested_reviewers_instructions will handle that part for you. [Get started with CodeRabbit](https://coderabbit.link/Q5oy5FI)

Introducing Change Stack: The first AI-native code review interface

Priyanka Kukreja — Wed, 13 May 2026 00:00:00 GMT

## **Code review has been broken for 18 years. We built the fix.** If you have been reviewing code for more than a few years, you know a quiet truth that nobody has bothered to fix. Code Review has essentially been the same since GitHub was launched in 2008. Files, alphabetical order, scroll to the bottom. Rinse and repeat 1,000 times a year. Meanwhile, the author who wrote the pull request understands it as a story, as a new data shape, the logic that consumes it, the call sites, and the tests. That story exists in their head but nowhere in the review interface. The reviewer has to reconstruct it from scratch, file by alphabetical file, before they can even begin to evaluate correctness. That cognitive overhead is why big PRs sit for days. It is why senior engineers often rubber-stamp anything over 500 lines. It is why architectural feedback is rare and nit-picks are abundant. The tool has been shaping the behavior, and not in a good direction. **Today, CodeRabbit is shipping Change Stack, a fundamentally different way of doing code review.** ## **What Change Stack does** %[https://youtu.be/yS0EwgA2zjw] Change Stack takes any pull request and reorganizes it from a flat file list into a guided walkthrough. Instead of presenting files alphabetically, it groups related work into a small number of independent change cohorts. Each cohort is broken into ordered layers that reflect the natural reading order of the change. It is the way a thoughtful senior engineer would choose to walk someone through it. And every layer anchors to specific line ranges in the actual diff, and carries its own AI-generated summary. When the structure of a layer warrants it, such as a new API contract, a state transition, or a cross-service call sequence, Change Stack generates a diagram alongside the diff. Change Stack renders the visual that best captures what the code is actually doing, whether that is a sequence diagram, a state machine, or an entity-relationship diagram. Not every layer gets one, just the ones for which a visual aid would make things easier to understand. ![Dark-themed code editor displaying a Git pull request with a side-by-side code diff.](/content/assets/change-stack-3panel.png) The three-panel layout puts cohort and layer navigation on the left, the diff in the center, and per-range context on the right. Keyboard navigation ('J' to advance, 'K' to go back, 'Z' for focus mode) keeps your hands off the mouse. Reviewers mark files as viewed, drop inline comments against specific ranges, accumulate a draft review, and submit a real GitHub review, all without switching tabs. Critically, reviews post back to GitHub natively. It is an upgrade to the review interface that still lands comments and approvals exactly where your team expects them, without disrupting your existing workflow. ## **Why you should care** If you are a senior developer or a tech lead whose team is shipping multiple, complex PRs in any given week, then you know that the *de facto* bottleneck on the team’s merge velocity is providing fast review without sacrificing quality. Change Stack directly addresses that constraint, by breaking down the code change into easy-to-review chunks, arranged in a logical manner. Change Stack also helps with cross-team reviews, those where you get pulled in because you had owned a touched file or a security-sensitive path. Prior to Change Stack, on a PR like this, you’d typically spend your first 20 minutes reverse engineering what the PR is even about before you can even consider next steps. Change Stack, however, makes that orientation immediate. For example, a Change Stack sequence diagram showing the new call path, or a state machine illustrating the updated lifecycle reduces that 20 minutes of reverse engineering to 30 seconds. ## **Why we think this approach is better** The current generation of AI code review tools address the [review bottleneck](https://www.coderabbit.ai/blog/code-review-needs-independence) by layering a smarter comment bot on top of the existing GitHub interface. But, that approach backfires. More comments on an already noisy PR thread increase the cognitive load on the reviewer, who now has to parse code changes, human feedback, and AI-generated commentary all at once. The problem is worse for large, unstructured diffs, and this problem has compounded sharply as AI-generated code volume has surged over the past year. Change Stack, however, is a completely different angle on how the review process works. It assists the reviewer to reduce this cognitive load by providing a walkthrough of how the PR author would have crafted the diff, all while augmenting that with visuals. ## **Built for how reviews actually work** ![Code review interface showing a highlighted comment extending IIRRReviewState with viewerSubmittedReview.](/placeholders/change-stack.png) * **Reviewable layers:** Change Stack breaks a PR into functional layers, so reviewers can move through the change by intent instead of the raw file order. * **Diagrams (when they help):** Layers can include Mermaid diagrams for real flows, state changes, data relationships, or schema changes. Simple layers stay text-only. * **Layer-scoped diffs:** Each layer shows the code ranges that belong to it, with surrounding context, so reviewers stay focused on the part of the PR being explained. * **Active summary sync:** As reviewers scroll through a layer, the right rail of the user interface highlights the summary that matches the code currently in view. * **Snapshot history:** Change Stack retains a snapshot for a PR, and reviewers can switch between them from a snapshot selector. * **Out-of-date warnings:** If the PR has moved beyond the snapshot being viewed, Change Stack shows that the view is stale and identifies the commit it was generated from. * **GitHub-native reviewing:** Comments and final review actions (approvals, change requests) sync with the GitHub surface. * **Public read-only sharing:** Public PR walkthroughs can be opened without signing in. Commenting still requires GitHub authentication. * **Reviewer-owned sign-in:** The GitHub sign-in flow preserves the exact Change Stack URL, including the selected layer or snapshot, and posts review activity as the reviewer. * **Opt-in workflow:** Change Stack opens from a CodeRabbit link and leaves the normal GitHub PR workflow intact for anyone who does not use it. ## **What comes next** Change Stack is the beginning of a more guided review experience. Code review should show more than what changed. It should help teams [understand why it changed](https://www.coderabbit.ai/guides/ai-agent-explainability), where to focus, and how each decision fits into the larger system. We are building toward reviews that feel less like parsing diffs and more like following the story of the software as it evolves. *The diff has not changed in 18 years. We think it is time.* **Change Stack is available for free for a limited time to all users. Try it on the next large PR that lands in your queue. The “Review Change Stack” button is in CodeRabbit's PR summary comment.**

Nobody is going to read the code

David Loker — Tue, 12 May 2026 00:00:00 GMT

Let me say something that's uncomfortable coming from someone in my position. AI coding tools are still way worse than humans at generating correct code. And humans were bad enough as it was. We're not even at the point where AI matches human error rates. We’re well past them in the wrong direction. That's not a reason to stop using AI coding tools. It's a reason to be honest about what comes next. Here's my prediction. Within the next 12 months, most developers are not going to be looking at the code at all. Let me explain why. ## The review problem is real, but it's not new Human code review has been breaking down for years. I used to spend 20-30% of my time reviewing code. That felt like a lot. It wasn't sustainable even then. Now AI-assisted development has accelerated the volume of code coming through review without adding any time to review it. The pressure that was already there is now acute. That part of the story has been told. What hasn't been told precisely is why the specific problems AI introduces are ones human reviewers are structurally bad at catching, and why that mismatch is what will make human review untenable, not just strained, within the next year. ## What human review can't keep up with The data makes this uncomfortably specific. Logic and correctness problems are 75% more common in AI PRs than human ones, according to CodeRabbit's [State of AI vs. Human Code Generation Report](https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report), which analyzed 470 open-source GitHub pull requests. Error and exception handling gaps are nearly double. These are the failure modes that cause outages, and logic errors are among the most expensive to fix and the most likely to cause downstream incidents. Catching them requires a reviewer to mentally exit the happy path and think through every edge case in code they didn't write and may not fully understand. ![Bar chart titled 'AI produces more critical issues' showing 341 for AI and 240 for Human.](/content/assets/what-human-review-can-t-keep-up-with.png) [Security](https://www.coderabbit.ai/blog/security-at-ai-speed) compounds this further. AI PRs show security issues up to 2.74x higher than human PRs. The most prominent patterns involve improper password handling and insecure object references. These aren't bugs you spot at a glance. Catching them means thinking like an attacker, asking not "does this work?" but "can this be exploited?" That's a different cognitive mode, and not one every reviewer can sustain reliably, especially across a large diff they didn't author. There's also a readability problem that makes all of this worse. AI-generated code produces more than 3x the readability issues of human code. That’s because it looks clean while quietly violating local conventions and structure. As the report puts it, AI-produced code often looks consistent but violates local patterns around naming, clarity, and structure. That's the worst kind of review problem, one where code that scans fine but requires deep familiarity with the codebase to catch what's actually wrong. And if you've been using AI code generation heavily, you may genuinely not know the choices that led to the code in front of you. The reasoning is opaque. You're reviewing an output, not a thought process. You don't know where to look because you don't know where the decisions were made. This is where the conversation about AI and code quality tends to stop. I think it needs to go somewhere else entirely. ## Verification, not just review What I see happening over the next 12 months is that validation shifts from reading code to [verifying intent with Change Stack](https://www.coderabbit.ai/blog/introducing-change-stack-the-first-ai-native-code-review-interface). Did the thing I wanted to happen, happen? Does the feature behave the way the prompt described? Those are answerable questions that don't require a developer to parse every exception handler in a sprawling diff. The layer that actually reads the code becomes automated. [AI code review](https://www.coderabbit.ai/), static analysis, security linting, and required test coverage are not stopgaps. The organizations already figuring this out are treating automated review as a structural requirement, not an optional layer. They're building validation pipelines that check outputs against intent. They're not debating whether the code looks right. They've accepted that they won't be the ones looking. Every company I talk to using AI code generation is viscerally aware of what's happening inside their organization. The excitement was real. The speed gains were real. So is the reality that sets in after a while. That reality is filled with reverting PRs, chasing bugs that are hard to attribute, and fixing things that should have been caught earlier. The amount of context a model would need to hold simultaneously to generate truly correct code makes perfect output a hard problem to solve quickly. These systems are not going to stop introducing issues anytime soon. We’re going to keep coding with AI. We’re addicted. So now comes the question of whether you have the infrastructure to catch what AI gets wrong before it reaches production. Most teams don’t have that infrastructure, and the gap between AI's speed and the ability to validate its output is widening. ![Green number +11,885, red number -26, and a progress bar on a dark background.](/content/assets/10k-line-pr.png) You can't review a 10,000-line PR the way you'd review a hundred-line one. Nobody can. The sooner engineering organizations accept that and build accordingly, the better positioned they'll be when, and not if, the human review model breaks down entirely. The code will keep coming. The question is what you put between it and production.

Introducing CodeRabbit Reverse Tunnel: AI code review for private-network enterprises

Henry Lau — Fri, 08 May 2026 00:00:00 GMT

[CodeRabbit Reverse Tunnel](https://docs.coderabbit.ai/self-hosted/coderabbit-reverse-tunnel) is a new private-network connectivity option, a first-party component that lets CodeRabbit review pull requests on a [GitHub Enterprise Server (GHES)](https://docs.coderabbit.ai/platforms/github-enterprise-server) instance, self-managed GitLab, and Gerrit that do not accept inbound network connections. If your team runs GHES inside a private network with no public endpoint, no inbound firewall exceptions, no vendor IP allowlisting on the GHES side, and no PrivateLink or peering path, this is for you. It's available for [Enterprise Plan](https://www.coderabbit.ai/enterprise) customers today. What it unlocks: * AI code review on a private Git platform, with no exposure to the public internet. * No new inbound firewall rules, no platform-side IP allowlist, and no PrivateLink or peering required. * A Connector that deploys into your existing container runtime, alongside other internal workloads. * A standard, auditable network shape: Outbound HTTPS on TCP 443\. ## Why we built Reverse Tunnel A class of enterprise customers operates GitHub Enterprise Server inside a private network with no inbound connectivity by design. These security constraints are typically codified in audit and compliance frameworks and cannot be relaxed for individual vendor onboarding. Existing solutions like VPN tunnels, public-ELB exposures, and PrivateLink peering require either an inbound route into the customer network or a cloud-provider solution. Without an alternative transport, these customers have no compliant path to deploy CodeRabbit. CodeRabbit Reverse Tunnel exists to close this gap. It enables these enterprises to adopt AI code review while preserving the security and compliance constraints they already have in place. ## How CodeRabbit Reverse tunnel works ![CodeRabbit Reverse Tunnel architecture diagram details secure data flow between customer and CodeRabbit systems.](/content/assets/architecure.png) CodeRabbit Reverse Tunnel is built around two components: a Connector that runs inside your network and a Gateway operated by CodeRabbit. Together they create a single outbound channel over which CodeRabbit can read from and write to your private GHES instance, without your network accepting any inbound connections. ### Components 1. **Reverse Tunnel Connector.** A lightweight, stateless container that runs inside your enterprise network. On startup, it dials out and establishes a long-lived WebSocket Secure (WSS) session to the Gateway on TCP 443\. All subsequent CodeRabbit traffic to GHES flows through this session. 2. **Reverse Tunnel Gateway.** A CodeRabbit-managed edge service that accepts Connector sessions and brokers CodeRabbit's runtime requests back through them. The Gateway authenticates each session using credentials issued to your tenant and routes traffic to the right Connector. ## **What happens when a developer opens a PR!** ![Sequence diagram illustrating a developer workflow with customer network and Lucihaus systems.](/content/assets/image2-12.png) When a developer opens a PR, GHES sends a signed webhook outbound through your existing NAT to CodeRabbit. Everything that follows runs on the WSS session your Connector has already opened: the Reverse Tunnel Gateway forwards CodeRabbit's read requests over WSS, the Connector translates them into internal HTTPS calls to GHES, and responses stream back the same way. CodeRabbit generates the review and posts the comments back through the same tunnel. ## **Getting started with CodeRabbit Reverse Tunnel** CodeRabbit Reverse Tunnel is available now for Enterprise Plan customers running GitHub Enterprise Server and GitLab in private networks. Reverse Tunnel now supports Gerrit, with support for additional Git platforms on the roadmap. Our goal is always to provide flexible deployment options for our customers and this release is a testament to that. Customers who self host with strict requirements can now take our state of art review engine to deliver high quality code faster. We’re committed to making our customers successful and our team is here to answer any questions you have along the way. For detailed information, please consult the [documentation](https://docs.coderabbit.ai/self-hosted/coderabbit-reverse-tunnel) or [contact our team](https://www.coderabbit.ai/contact-us/sales).

How we built the CodeRabbit plugin for Codex

Juan Pablo Flores — Tue, 28 Apr 2026 00:00:00 GMT

When we started working on the [CodeRabbit plugin for Codex](https://www.coderabbit.ai/blog/coderabbit-plugin-for-codex), the goal was not to package as many features as possible. It was to make one workflow feel natural. A developer asks for review, the agent handles setup and execution, and the feedback comes back inside the same working session. We wanted that experience to live inside the surfaces where developers are already coding, so review becomes something they reach for in the moment rather than a separate step they have to context switch into. Getting there required more than writing a short set of instructions. We had to decide what belonged in the plugin, how much of the workflow the agent should own, and how explicit we needed to be about model behavior for the experience to stay reliable. ![Flowchart showing Before and After code review workflows, from external steps to inline feedback.](/content/assets/context-switch-vs-seamless.svg) ## **Start with the workflow** We started with the user outcome, not the package structure. The question was straightforward: what should become easier once the plugin is installed? For us, the answer was that code review should feel like part of the coding flow itself. A developer should be able to ask Codex to review the current changes, or use `@coderabbit` to invoke the plugin directly, and get a useful result without manually checking setup, switching tools, or reconstructing the right review command. That workflow gave us the shape of the plugin. Instead of designing around a long list of capabilities, we designed around a narrow job to be done and then asked what the agent needed to do that job well. The more of that loop we could keep inside the place where developers were already coding, the more we could reduce context switching, shorten review cycles, and make code changes cheaper to apply. ## **Build with focused skills** From there, we built focused skills around the core review experience. The code review skill does the heavy lifting. It verifies that the [CodeRabbit CLI](https://www.coderabbit.ai/cli) is available, checks authentication, chooses the right review target, runs the review, and summarizes findings by severity. Splitting the workflow into focused skills was an important design choice. One giant instruction file might seem simpler at first, but it quickly becomes harder to maintain and harder for the model to use consistently. Focused skills keep the behavior clearer, make iteration easier, and give us a cleaner way to add new workflows over time. ![A diagram comparing a single large instruction file to a modular plugin approach with focused skills.](/content/assets/monolith-vs-focused-skills.svg) **The plugin system's core advantage is that it lets you compile skills, MCP servers, and connectors into a single installable unit.** For teams building developer tools or services, that is a meaningful improvement to the experience of the people consuming your work inside the Codex app. Instead of asking users to discover, install, and remember the name of each individual skill, you can ship everything together in one plugin. The model then decides when to bring each skill into the conversation where it provides the most value. **That reduces cognitive load for developers and makes the whole experience feel more intentional.** As we add new [CodeRabbit skills for Codex](https://docs.coderabbit.ai/cli/codex-integration), users get them through the plugin instead of returning to install one skill at a time. A good plugin does not have to be large. It has to make one important workflow easier, then create a clean path to expand from there. ## **Design for real model behavior** The most important part of the work was not the packaging itself. It was learning how to write skill instructions that guide the model toward the behavior you actually want. Every plugin builder will go through a version of this process, so here are the lessons that made the biggest difference for us. **Be explicit about tool choice.** Models are resourceful, and that resourcefulness can work against you if the skill does not set clear boundaries. Early on, we noticed Codex reaching for Python when the workflow only needed a direct CLI command. It would wrap CodeRabbit in a script, add layers around simple terminal actions, or introduce setup steps that were not needed. Once we made the skill instructions specific, telling the model to run `coderabbit` directly as a bare shell command and not to use Python wrappers, the behavior became consistent. The lesson: if your plugin depends on a particular tool, say so clearly and close the door on alternatives the model might improvise. **Handle authentication as a first class concern.** When the CodeRabbit CLI was not yet authenticated, the model would try to solve that on its own rather than following the guided path we had built. It might skip the auth step, guess at credentials, or improvise a workaround that looked reasonable but did not actually sign the user in. We initially tried using the authentication flags that the Codex team provides in their documentation, but in our experience and from the developers testing it, we did not see meaningful changes in behavior when we implemented them. It is possible we configured something incorrectly, but the approach that actually made the difference was handling it ourselves in the skill: check authentication status early and, when the user is not signed in, fall back to a step by step flow that walks them through setup. That one change eliminated most of the unpredictable behavior we were seeing on first runs. ![Flowchart illustrating a review process with authentication, setup flow, and summarization of findings.](/content/assets/auth-first-flow.svg) **Set expectations for long running tasks.** CodeRabbit can take time to analyze a larger set of files, and without guidance the model can interpret that delay as a sign something has stalled. We saw it stop early, retry too quickly, or move into a fallback path before the review had actually finished. The fix was to be explicit about patience in the skill: let the review run, wait through the full timeout window, and only narrow the scope after a genuine timeout rather than treating normal latency as a failure. If your tool has operations that take more than a few seconds, building that expectation into the skill makes a real difference. **Guide communication style.** During long tasks, the model tends to narrate every step, repeat that it is still waiting, and send updates that add more noise than reassurance. Users want to know the plugin is working, but they do not want a stream of status messages competing for their attention. We addressed this by telling the model to stay quiet during reviews and only speak when user input is needed, the review is complete, or an error requires attention. The result was a calmer, more professional experience that users consistently preferred. **Design for both the app and the CLI.** We initially built the plugin for the Codex app, and when we moved to testing it in the CLI we noticed different patterns emerging. One of the major benefits of the Codex app is its ability to render UI directly. Codex can display markdown, tables, and richer formatting that makes review findings easy to scan. But the CLI does not render tables or more complex UI components the same way, and what looked clean in the app became harder to read in the terminal. We had to go back to simpler primitives to make sure the output worked well in both environments. If you are building a plugin that will run across the Codex app and the CLI, it is worth testing both early and designing your output around the more constrained surface first. ## **What we would recommend to other builders** If you are building your own Codex plugin, start with the user outcome and work backward from there. Ask what should become easier once the plugin is installed, then define the smallest set of skills that supports that outcome well. The lessons above on tool choice, authentication, patience, and communication style all came from that same process of working backward from the experience we wanted and then writing the skill instructions to get there. **The Codex app includes a create plugin skill** that can help you scaffold the structure and get everything set up. It is a useful way to get moving without assembling the pieces from scratch, and it gives you a working starting point that you can iterate on as you learn how the model responds to your specific workflow. The technical integration is only one layer of the work. The deeper design challenge is deciding what the model should do by default so the experience feels intentional from the first run. ## **What comes next for CodeRabbit Codex plugin** This first version gives us a base to keep expanding the CodeRabbit experience in Codex. We plan to keep improving how review feedback flows back into the agent, add new skills over time, and continue tightening the first run experience so users reach value faster. One of the ideas the team is most excited about is exploring how to use not only the context of the code itself but also the conversation context that Codex provides through the messages a developer has exchanged during the session. That history carries a lot of signal about the intention behind a set of changes, and feeding that into the review could help CodeRabbit deliver feedback that is more aligned with what the developer is actually trying to accomplish rather than reviewing the code in isolation. If you want to try the plugin, head to the announcement post for installation steps. If you are building your own, we would love to see what you create. Share it with us in the [CodeRabbit subreddit](https://www.reddit.com/r/coderabbit/) or the [CodeRabbit Discord](https://discord.com/invite/coderabbit).

What the Vercel breach means for enterprise code security

Sehtej Khehra — Fri, 24 Apr 2026 00:00:00 GMT

Just days ago, Vercel, a widely used cloud platform for deploying web applications, [disclosed](https://vercel.com/kb/bulletin/vercel-april-2026-security-incident) a breach that began months earlier. It started in a roundabout way, when a Context.ai employee unwittingly installed a Lumma Stealer disguised as a Roblox script. A Vercel employee, using Context.ai, then unwittingly got caught up in this trap. The attackers harvested the Vercel employee’s Google Workspace credentials via stolen OAuth tokens, then moved laterally and breached Vercel's internal systems. This exposed API keys, tokens, database credentials, signing keys, and more. In response, Vercel advised customers to rotate any environment variable not marked "sensitive" and to treat those values as compromised. The inevitable post-mortems of this hack will focus on OAuth governance and third-party SaaS risk. Those angles are valid, but they miss the point for security leaders responsible for the code itself: *This was a developer supply chain attack, and the stolen assets prove it.* ## **What the Vercel breach taught us, and how CodeRabbit is built to address it** Our customers trust us with their most valuable asset, their source code. That trust is why security isn't an afterthought. It's a CodeRabbit design principle. ### **Lesson 1: Every tool that touches your code is part of the attack surface** The right question isn't whether a component in your developer stack gets breached. The better question is, what's the maximum damage an attacker can do from that point? ![CodeRabbit Architecture diagram detailing Handler, Reviewer, Database, Scanner, Token Service, and Audit Vault components.](/content/assets/security-posture-cr.png) We built the [CodeRabbit code-review platform](https://www.coderabbit.ai/) around that question. Every code review runs in an isolated, ephemeral sandbox and is provisioned per event and destroyed after completion. Each sandbox holds a single, short-lived token scoped only to the repository under review. There’s no shared state between customers. There are no long-lived credentials. There’s no access to internal networks. Sandboxes can reach the public internet when tools require it, but cannot reach CodeRabbit's internal services. Stored code is encrypted with per-customer keys, inaccessible even to CodeRabbit employees. The result is this: If a sandbox is compromised, there's nothing to pivot to. There’s no persistent tokens, and no lateral movement paths. *If one of your sandboxes or workers is breached tomorrow, what's the worst-case outcome?* Every enterprise should ask this of every vendor in their developer stack. ### **Lesson 2: Long-lived secrets in code are a major risk** Many Vercel customers had to rotate keys they didn't know were exposed because the most damaging credentials are often the ones teams forget exist, buried in environment variables or hardcoded directly in source files. Code review is the last practical checkpoint before a secret becomes permanent. Once a credential is committed to a Git repository, it can't be fully scrubbed. Copies persist in forks, caches, CI logs, and developer machines. The only reliable defense is catching it at the pull request. CodeRabbit flags hardcoded credentials through a combination of pattern matching and AI-powered contextual analysis that understands data flow. Pattern matching catches formats like `sk_live_*`, `AKIA[A-Z0-9]{16}`, `ghp_[a-zA-Z0-9]{36}`, and variables named `*_SECRET`, `*_KEY`, or `*_PASSWORD`. We also integrate tools like Semgrep, Checkov, Brakeman, and Betterleaks, with one-click fixes surfaced directly in the pull request. Security teams can define custom checks in natural language via `.coderabbit.yaml` and enforce them as pre-merge gates. Examples include blocking files that hardcode database DSNs or flagging OAuth scopes broader than `read:user`. Vercel has since updated its platform so new environment variables default to sensitive. That's a step in the right direction, but it only covers secrets that make it into environment variables. It doesn't catch credentials hardcoded in source files, feature branches, comments, or config files. The more robust approach: treat every credential as sensitive by default, and enforce that at the code review layer before it ever reaches production. ### **Lesson 3: Identity and access controls must extend to the code review layer** At its core, the Vercel breach was an identity breach. An OAuth token issued to a third-party app became an attacker's access path. Every tool with OAuth access in your workspace, every CI service running on long-lived GitHub tokens, every AI assistant with read access to a monorepo, each of those is a potential entry point. Your code review platform deserves the same identity rigor you apply to your identity provider. For [CodeRabbit Enterprise](https://www.coderabbit.ai/enterprise), that includes: * SSO and SAML support across major identity providers, with enforcement rather than optional bypass * Custom RBAC to enforce the principle of least privilege * Audit logging for all platform actions and administrative changes * Self-hosted deployment options for organizations that require code to remain within their network perimeter * Support for multi-organization structures, including subsidiaries and M\&A scenarios * SOC 2 Type II compliance, validated annually through independent audit * Zero data retention after review completion, with all code disposed of once the review ends * Vendor security reviews as a standard part of procurement *The goal is straightforward. Even if something upstream is breached, your code review tool should never be the next entry point in the attack chain.* ## **Questions enterprise security teams should ask vendors selling developer tools** The Vercel breach is a reminder to reassess every tool that touches your codebase. Ask these questions of any vendor with access to your source code, including us: 1. **Sandbox isolation:** Is it per-event, per-customer, or shared? What's the worst-case outcome if one sandbox is compromised? 2. **Token scope and lifetime:** Are tokens short-lived and repo-scoped, or long-lived service credentials? 3. **Encryption at rest:** Is code encrypted with per-customer keys? Can your own employees read stored customer code? 4. **Data retention:** Is code disposed of after review, or cached indefinitely? 5. **Self-hosted option:** Can organizations that can't send source code to an external SaaS deploy on their own infrastructure? 6. **SOC 2 and pen testing:** When was your last third-party security assessment, and is the report available under NDA? 7. **Vulnerability Disclosure Program:** Do you have one, and how have you responded when a researcher reported a finding? 8. **Sub-processors:** What third parties handle your data, is the list public, and how much notice do you give before adding a new one? 9. **Identity controls:** Do you offer SSO enforcement, SAML, SCIM, RBAC, and audit log export to a SIEM? 10. **Breach response:** How will you notify customers, how quickly, and what will you disclose? CodeRabbit's answers are detailed in our [full security architecture](https://www.coderabbit.ai/blog/our-security-posture-how-we-safeguard-your-repositories), and in our [Trust Center.](https://trust.coderabbit.ai/) Our team is also happy to answer any questions. ## **At CodeRabbit, security is our top priority** Supply chain attacks don't start at the primary target. They start at the weakest link. Every vendor in your developer workflow, every tool with a token, an OAuth grant, or read access to your code is a potential entry point. The Vercel breach didn't begin at Vercel. It began with a Roblox script. Demand answers. Every vendor you trust with your source code should be able to tell you exactly what happens if they're the next Vercel.

What changed in OpenAI GPT-5.5: better judgment, stronger coding, better signal

Juan Pablo Flores — Thu, 23 Apr 2026 00:00:00 GMT

*Our early testing shows the model communicates more directly, finds higher-signal issues, and performs better in practical coding and review workflows.* *Note*: During release, you can try GPT-5.5 in ChatGPT and Codex. Using GPT-5.5 felt different in a fairly specific way: it was quicker, leaner, and more direct. In practice, that meant shorter responses, more selective review behavior, and a stronger bias toward small workable changes instead of broad rewrites. This is most relevant for teams using CodeRabbit, building agent workflows, or asking the model to make real changes in codebases. The model responded quickly, communicated with less overhead, and moved toward practical answers without much wasted motion. Part of that speed was also visible in how quickly it surfaced user-facing progress instead of waiting to finish all of its internal work before responding. That directness carried over into the quality of the output. Across code review, bug-fixing, and debugging tasks, the model consistently leaned toward scoped changes, preserved behavior more often, and usually focused on the actual failure mode rather than drifting into speculative redesign. ## **GPT-5.5 code review performance** One of the clearest strengths in our testing was code review. In the stronger examples, the model focused on bugs that were concrete, actionable, and worth interrupting a developer's flow. It showed up clearly in debugging-oriented reviews. When the task involved access control, error handling, or API behavior, the model was often able to isolate the actual regression, reject a weak diagnosis, and point toward a fix that preserved the intended behavior. The benchmark results align with this observation. The "baseline" mentioned in the subsequent diagram and text does not refer to a single comparison model. Instead, it signifies CodeRabbit's current live review system, which employs a combination of various models rather than relying on just one. In our early testing with GPT-5.5, the agent reached 79.2% expected issue found on our curated review benchmark versus 58.3%, improved precision from 27.9% to 40.6%, and produced 75 comments versus the baseline's 67\. That means it found substantially more useful issues with only a modest increase in comment volume. In testing, GPT-5.5 improved performance across several key metrics on our large-scale, real-world review set. Specifically, the expected issue found rate increased from a 55.0% baseline to 65.0%. Furthermore, precision improved from 11.6% to 13.2%. The agent also became more verbose, generating 722 comments compared to 558 with the previous baseline. The practical takeaway is that GPT 5.5 outperformed the baseline on signal quality, even though its review volume was not uniformly lower than baseline. ![Bar charts comparing GPT 3.5 and a production baseline for review signal quality metrics.](/content/assets/image2-3.png) Part of that behavior appears to come from the review standard behind the prompt. In plain terms, the model is being pushed to flag bugs that are real, local to the change, and specific enough that the author could fix them. It is not supposed to guess about hidden intent, complain about broad codebase quality, or flag issues that a compiler, type checker, or linter would already catch. The associated comment guidance follows the same logic: comments should be brief, explicit, and matter-of-fact. If this performance carries into a future rollout, it would be a tangible benefit for CodeRabbit users. Increased selectivity translates to reduced time spent sifting through irrelevant comments, fewer redundant review threads, and a greater likelihood that the feedback you receive highlights the issues most worth addressing. ## **GPT-5.5 coding performance** Separate from review, the model performed well at code generation and implementation work. Its main strength here was control, as it tended to choose scoped changes, preserve existing interfaces when asked, and avoid overbuilding. The examples collected show that pattern clearly. When tasked with extending endpoints, maintaining route contracts, or resolving operational issues, the model consistently favored precise modifications with predictable results: enhanced safety, interface preservation, and minimized unintended consequences. Rather than completely rewriting the code, it prioritized the smallest possible modification to resolve the issue while maintaining the stability of the surrounding system. GPT-5.5's code generation excels when tasks are specific and limited in scope. It is highly effective for focused tasks such as bug fixes, minor API adjustments, refactoring that maintains original behavior, and adding targeted tests. In UI tasks, that same pattern produced polished interaction work and solid library use, though originality remained more limited than execution quality. Our team saw better-than-expected animation handling and unusually detailed interactions, but also a tendency to fall back on familiar styling choices, including a noticeable indigo-violet color bias. ## **GPT-5.5 works best with clear direction** For developers using the model directly, the interaction pattern is fairly clear. The model appears to do best when the task is scoped, the constraints are explicit, and the environment can provide feedback. In practice, that means giving it concrete requirements, preserving interface expectations, and letting it run or inspect the system when possible. The model performed better when it could move through a visible loop of change, inspection, and correction rather than trying to solve everything in one shot. That fits the broader pattern we saw throughout the tests: more direct output, less wasted tokens, and better results on bounded tasks. Our team's testing revealed that the model followed instructions too literally, especially when the prompt was poorly structured, lacked detail, or had weak underlying concepts. In those cases, the model often did not repair the direction on its own. It tended to execute the request as written, even when a more experienced collaborator might have paused, clarified, or challenged the premise. ![A product analysis slide detailing GPT 5.5's strengths and limitations in coding work.](/content/assets/image1-17.png) That means the prompt quality has a bigger effect on the result than developers expect. The model looks strongest when the request is specific about the intended behavior, the constraints, and the success criteria. A vague or internally inconsistent prompt can lead the model to generate a rapid response, but that output is more likely to mirror the prompt's existing weaknesses rather than correct them. ## **Reduced tokens for long-running agents** A noteworthy focus of GPT-5.5 is efficiency. While this is harder to measure with benchmarks than performance in reviews, it emerged as one of the most distinct trends in our testing. The model was often less verbose and surfaced visible progress quickly. This suggests the new version requires fewer tokens for equivalent tasks compared to previous models, a benefit that is difficult to isolate in a single benchmark figure. This is especially relevant for agent harnesses that depend on repeated iterations to converge on the right answer. In systems that follow OpenClaw-like patterns, or any workflow where an agent has to plan, act, inspect, retry, and refine over many cycles, token inefficiency compounds quickly. If the model can stay concise while still being effective, it can reduce the amount of token-heavy overhead in those longer loops. For teams building on external services or agent platforms, that can mean more room for iteration before token usage starts to drag down the workflow. ![Chart illustrates GPT 5.5's superior token efficiency compared to a verbose model, enabling more retries.](/content/assets/image3-6.png) ## **Should you use GPT-5.5?** GPT-5.5's primary strengths lie in the developer workflow, specifically excelling at: identifying more substantive issues than the previous model, implementing focused changes without broad refactoring, and effectively self-correcting after initial errors. For teams evaluating tools like CodeRabbit, the more defensible claim is better signal per review than baseline, not universally lower review volume. For developers using the model directly, the pattern is just as clear: give it a scoped task, make the constraints explicit, and let it verify its work against the actual system. [Get started with CodeRabbit](https://coderabbit.link/NcmblIe), connect your repo, get your first AI review in minutes. Free to try, no credit card required.

Your AI agent has amnesia

Harjot Gill — Wed, 22 Apr 2026 00:00:00 GMT

\> **Schedule disasters and system bugs arise because the left hand doesn't know what the right hand is doing – Fred Brooks in [*The Mythical Man-Month*](https://en.wikipedia.org/wiki/The_Mythical_Man-Month) describing the [IBM OS/360 Project’s](https://spectrum.ieee.org/building-the-system360-mainframe-nearly-destroyed-ibm) coordination failures that nearly sank it.** ![Text discussing communication issues and their consequences in large programming projects.](/content/assets/communication-program-project.png) The *Mythical Man-Month* remains one of the most popular books in engineering management and a staple on leaders’ shelves. Fifty years of SDLC evolution has been one long argument against that failure mode. [Agile](https://agilemanifesto.org/) placed developers in the same room as the product. DevOps tore down the wall between dev and ops. [Git](https://git-scm.com/book/en/v2/Getting-Started-A-Short-History-of-Git) and pull requests gave teams a shared record of what changed and why. [CI](https://martinfowler.com/articles/continuousIntegration.html) made the build everyone's problem. Platform teams built paved roads so tribal knowledge wouldn't have to be re-earned by every new hire. Every major shift was a bet on *shared understanding* beating *individual heroics*. Then coding agents arrived, and we regressed. Every engineer now runs a private agent, on their own machine, in sessions nobody else can see. The agent starts each morning knowing nothing. Whatever reasoning was assembled yesterday, whatever alternatives were weighed, context loaded, decisions made in standups, are gone. The process starts to break, creating a false equivalence, “We are shipping faster than ever\!” ![Bar chart showing relative percentage changes in developer metrics with AI's influence.](/content/assets/coderabbit-agent-1.png) Brooks was describing human teams drifting apart over weeks. We're now adding AI teammates who drift apart within a single workday, and calling it high productivity. A new hire at least retains what they learned yesterday. ## The context tax Recently, I watched a senior engineer spend 11 minutes setting up context before his coding agent wrote a single line. He described the architecture, and explained why they use Postgres and something else. Finally, he pasted three files, a Linear ticket, and a Slack message of context from his tech lead about a service being deprecated. ![Diagram showing developer workflow from various tools to an agent-generated code output.](/content/assets/screenshot-2026-04-21-at-3-21-10-pm.png) The agent wrote some code, pretty good code, actually. The senior engineer opined in the end, the next morning, that he would open a new session and probably do the whole thing again for a different context, from scratch. The agent lacks the team level insight. This is AI-assisted development in 2026\. ## The wrong problem Nobody is talking about what the agent knows *before* it starts working. Right framing of a problem makes an average thinker perform like a brilliant one, at least for humans. For AI agents, the gap is more substantial. The same model, given the same prompt, will produce wildly different code depending on whether it understands the system it's working in. Give an agent your codebase with no context and you get plausible-looking code that misses every convention your team spent two years establishing. Give the same agent your codebase plus your architectural decisions, ticket history, on-call runbooks, and the Slack thread where your team decided to deprecate the old auth service, and you get code that looks like a teammate wrote it. Right now, developers are the ones manually assembling that context every single session, acting as interpreters between their own engineering org and a tool that should already know. ## Five kinds of agent amnesia The context problem is the most visible symptom, but the disease is more systemic. There are at least five ways the current generation of agents forget: ### Systems Every session starts cold with no context on systems and choices. The agent opens a file and has no idea why your team chose Postgres over Redis for queues, which auth service is about to be deprecated, or that you decided last quarter to let errors bubble up unwrapped. However, that context exists in a PRD nobody reads, a Slack thread from March, and a senior engineer's head. The first thing a developer does every morning is translate it into a prompt. Again, the dev is reconstructing information that already exists somewhere else in your company, and is adding it into a context window that is limited to a personal session. ### Its own past work The agent you taught yesterday isn't the agent you're using today. You walked it through the billing module: the retry logic that looks wrong but isn't, why you can't trust Stripe's idempotency key on webhooks, and the failed refactor from last summer that's still in the git history as a warning. It produced good code. You closed the tab. What's left is a commit. The reasoning that led to it (the part you'd actually want to keep) went away with the session. Open the same file next week, and you'll do it all again. PS: Yes, you are storing/sharing in [agents.md](http://agents.md) or another durable knowledge base. But is that continuously updated enough to gather crucial team context to make it easily maintainable for the team's day-to-day work? ### Teammates Coding agents are single player. One developer, one session, one machine. The work is invisible to everyone else on the team. If you need to hand off something, your teammate starts from zero because there's nothing to hand off *to*. We spent a decade making development collaborative: Git, pull requests, shared CI, and dashboards that anyone can open. Then we built AI dev tools that made development siloed\! ### What happened next The agent writes a piece of code, makes it to PR, and goes through code review. Next sprint, someone rewrites half of that code and a month later there is a bug in the code that intermittently brings the production level down. Engineers then collaborate on Slack or GitHub to fix the regression. Your agent is out of the loop on all these decisions. Engineers get better by shipping code that gets reviewed, deployed, broken, and fixed. At each stage, there are collective learnings for a team and individual. If you remove the agent from that loop, you are left with an agent that generates plausible code indefinitely, without ever learning which forms of “plausible” actually survive contact with production. ### That somebody is watching Okay\! We are not gonna get past every developer running a local AI coding agent whether it is in CLI, IDE or other new form factors (ADE?\!) . However, the engineering org has no shared view of what and how agents are performing across the team. Which systems are they touching? How are they spending? The economics will force the question soon enough. Uber's CTO told [The Information](https://www.theinformation.com/newsletters/applied-ai/uber-cto-shows-claude-code-can-blow-ai-budgets) this year that his team had already burned through its entire 2026 AI coding budget. Agent amnesia is essentially becoming an economics question. Token spend multiplied by each engineer and compounded daily is a harder problem for engineering teams to solve because they can’t walk away from the productivity gains that got them here in the first place. ## But my terminal coding agent works fine For you. Individually. On your machine. Yes. I'm not arguing that terminal agents don't work. They clearly do. Claude Code, Codex, they're great tools. And, we're not trying to replace them. In fact, features like steering control, sub-agent framework, hooks, and plugins are all immensely useful for a developer, helping them produce more code than ever\! To be clear, these tools do have memory. Claude Code reads a project-level CLAUDE.md, a user-level one, and persists notes in a local memory directory that survive across sessions. It's the durable team knowledge that is lost in local agentic coding sessions. The senior engineer's hard-won understanding of the billing module lives in her markdown files, on her laptop. When a colleague across the geo opens their first session, none of it carries across or persists. ## What agents need to become Edsger W. Dijkstra [wrote](https://www.cs.utexas.edu/~EWD/transcriptions/EWD03xx/EWD340.html) that the competent programmer is fully aware of the strictly limited size of his own skull. Agents don't have skulls. They don't have to forget. We made them forget because we modeled them on ephemeral terminal sessions instead of durable environments where teams actually work. Here's what the next generation looks like, once we stop treating amnesia as a given: **Knowledge should compound.** Instead of a context file lingering on one developer’s machine, every code review, resolved ticket, and architectural discussion should feed a layer that makes the agent more useful next month than it is today. This is a shared knowledge layer, scoped by team, project and domain. When a new engineer joins, the agent already carries the institutional memory that probably lived in a few senior engineers' heads. **Work should persist and be resumable.** Start a task in a Slack thread. Get interrupted. Hand it to a teammate. Come back two days later, and the work is still there. It’s visible, commentable, and resumable by anyone with the right access. It’s not trapped in a browser tab on one developer’s laptop. **The agent should be governed like any other system with production access.** Most coding agents today are priced per seat and scoped per user. Your spend is whatever your developers happen to burn. Your access controls are whatever the tool defaults to, which is usually "everything." Your visibility into what the agent is actually doing across your org is approximately zero. You wouldn't give every engineer root access to prod. There's no reason to give an AI agent flat access to your entire org's context, either. **Context should be auto-assembled from everywhere, not pasted in by hand.** The agent should pull from your codebase, tickets, docs, observability stack, cloud infrastructure, and the knowledge your team has built up over time. The developer's job is to direct the agent, not to be its research assistant. The No. 1 complaint about coding agents is that they generate throwaway code. Too much rework and too many iterations before something is actually mergeable. Better context on the first pass, however, means better code on the first pass. ## **Where this goes** Today, we are introducing [CodeRabbit Agent for Slack](https://coderabbit.link/agent). We're starting with the core loop: planning, code generation, review, investigation, and knowledge-augmented development.One agent for your entire Software Development Lifecycle. It’s all inside Slack, synchronous, and having guardrails in place. But the vision is bigger than a coding agent in Slack. What we're building toward is an agentic layer across the entire SDLC, a layer that understands systems, connects tools, retains the team's knowledge, and executes engineering workflows end-to-end. * Post-deploy regression triage that correlates a Datadog spike to a specific commit and drafts the fix, without anyone having to manually assemble the context. * Breaking change detection that traces downstream consumers across repos before you merge. * Customer bug investigation that starts from a support ticket, checks your error logs, finds the root cause, and posts a preliminary analysis to the team. * Sprint digests that pull from Linear, GitHub, Datadog, and PostHog. We believe the agent is only as useful as the context it can reach and the actions it can take. *The endgame is the layer that connects the stack and makes it programmable from the place where your team already works.* ## **The bet** Individual productivity is up across every engineering team adopting AI. Team-level productivity is still stuck, and it's stuck for three reasons that have nothing to do with model quality: there's no explainable record of what the agent actually did, no cost attribution that matches how teams are organized, and the agent doesn't live where engineering actually happens. Solve those, and team productivity starts to compound the way individual productivity already has. The agents that win won't be the ones with the best model. They'll be the ones that already know your systems when you open them — the conventions, the deprecations, the decisions that didn't make it into a doc. That's durable team knowledge, and it's what's missing from every tool shipping today. I'm not going to tell you this replaces developers. If you've shipped production code, you know that the hard part of engineering isn't typing. It's judgment. It's understanding the system. It's knowing what to build and what not to build. At CodeRabbit, this is the bet we've been making for years. Our independent purpose-built context engine, running two million code reviews a week — is how we've encoded what good engineering teams actually do. CodeRabbit Agent extends that engine into Slack with your team's own decisions and patterns on top. Work happens in the open, where teammates can see it, jump in, and pick up where someone left off. Context compounds, rather than dying at the end of a session. ***Whoever solves agent amnesia wins. Whoever keeps building faster with amnesiac agents is building a faster version of the wrong thing.*** Sources: Figure 1: Sources: Becker, J., Rush, N., Barnes, E., & Rein, D. *Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity*. METR, July 10, 2025; Xu, F., Medappa, P. K., Tunc, M. M., Vroegindeweij, M., & Fransoo, J. C. *AI-assisted Programming May Decrease the Productivity of Experienced Developers by Increasing Maintenance Burden*. arXiv:2510.10165, 2025\.

What Claude Opus 4.7 means for AI code review

Juan Pablo Flores — Thu, 16 Apr 2026 00:00:00 GMT

You know the bug that ships on a Friday because the reviewer was rushing through a 40-file PR? The race condition buried three files deep that nobody traces until it pages someone at 2 AM? That's the gap AI code review was built to close. With Claude Opus 4.7, the gap just got a lot narrower. CodeRabbit's review engine doesn't rely on a single model. We run an ensemble of frontier models from multiple labs, selecting different models for different aspects of the review pipeline. Each model earns its slot through evaluation on real code. When a new frontier model ships, we benchmark it against every model in our current ensemble to see where it outperforms and where it doesn't. We've been testing it at CodeRabbit against our production code-review pipeline. The results aren't marginal improvements. We ran Opus 4.7 head-to-head across 100 evaluation points spanning a multitude of real-world open-source pull requests. Claude Opus 4.7 finds more real bugs, produces more actionable feedback, and reasons across files *better than anything we've tested before*. ## **How we evaluate models at CodeRabbit** Before diving into the results, it's worth understanding how we benchmark code-review models. Methodology matters as much as outcomes. Our evaluation framework is built around what we call Error Patterns (EPs): a curated set of 100 known issues drawn from actual pull requests across major open-source projects. Each EP maps to a specific, verified issue in a real PR: a race condition in a Go service, a missing null check in a React component, an authorization bypass in a Rails controller. For every model we test, we measure four core dimensions: 1. **Pass rate:** Does the model catch the known issue? 2. **Actionability**: Does the feedback tell the developer exactly what to fix? 3. **Comment Quality**: Does the model correctly classify severity? Is the output well-structured and code-backed? 4. **Signal-to-noise**: How much useful feedback does the model produce relative to noise? We scored Opus 4.7 against our current production baseline on the exact same rubric, across the same 100 EPs, on the same PRs. No cherry-picking, no special prompting for one model over the other. ## **Model performance on code reviews** Integrating Opus 4.7 in CodeRabbit delivers a jump in review quality across various metrics that we track. ![Performance comparison of AI code review: Claude Opus 4.7 significantly surpasses baseline metrics.](/content/assets/figure1-code-review.png) ### **Pass rate** On our core evaluation, whether the model catches the known issue in a given PR, integrating Opus 4.7 to CodeRabbit’s current code review harness passed on 68 out of 100 evaluation points, up from 55 on the baseline. That's a **24% relative improvement** in the model's ability to find the specific bug that matters. To put this in practical terms: imagine a team that merges 20 PRs a week, each containing at least one reviewable issue. With the baseline model, roughly 11 of those issues get caught. With Opus 4.7, that number jumps to nearly 14\. Over a quarter, that's roughly 36 additional bugs caught before they reach production. ### **Full-system score** When we layer in our full scoring system (which accounts for outside-diff context, nitpick filtering, and overall review coherence), the gap widens further. Integrating Opus 4.7 scored **74/100** compared to the baseline's **60/100**, a **23% relative improvement**. This metric captures something subtler than raw bug detection. A model might catch a bug but do so in a way that's confusing, references the wrong line, or buries the finding in unrelated noise. The full-system score penalizes those failure modes and rewards reviews that are coherent, well-targeted, and properly contextualized within the broader PR. The fact that Opus 4.7's full-system score improved *more* than its raw pass rate tells us the presentation quality improved alongside detection. The reviews are more coherent, better targeted, and properly contextualized. ### **Actionable review rate** Every single one of the 640 comments was marked actionable by our evaluator, meaning each one contained enough information for a developer to act on. But when we measure against EP-specific actionability (whether the actionable comment actually addresses the target issue rather than a tangential concern), **it jumped from 54% to 64%**. This is the difference between a reviewer who says "there's a problem somewhere in this file" and one who says "line 47 will panic when the user is nil because the guard clause on line 42 doesn't cover the `admin` role path. Here's a diff that fixes it." Both are technically actionable. Only the second one saves you time. ### **Important-issue yield** This is one of the most striking data points in our evaluation. Nearly **70% of all comments** Opus 4.7 generated were classified as important, meaning they flagged substantive bugs, security risks, or correctness problems rather than style nits or cosmetic suggestions. Of those 443 important comments, 367 were findings the model surfaced *beyond* the target evaluation point. That's 82.8% of all important output coming from issues the model discovered on its own, unprompted, while reviewing the same code. In other words, Opus 4.7 behaves less like a targeted test and more like a thorough reviewer who notices problems in the periphery while looking at the code you pointed it to. For context, the baseline model generated 558 total comments. Integrating **Opus 4.7 generated 640, about 15% more volume**. But the important-issue density is what sets it apart. More comments don't matter if they're noise. More *important* comments are a different story entirely. ## **What makes Opus 4.7 different under the hood** The scores above establish that Opus 4.7 is better. What follows explains *why*, and what it actually looks like when this model reviews your code. We spent significant time reading through individual comments, and several patterns emerged consistently across languages and codebases. ![Presentation slide highlighting Opus 4.7's Deep Bug Detection, cross-file bug connection, and fix-shipping reviews.](/content/assets/figure-2-different.png) ### **Deep, mechanism-level bug finding** Across our evaluation set, the model consistently identified concrete races, nil/panic paths, authorization failures, blacklist bypasses, XSS and SSRF chains, response-shape mismatches, and lifecycle/data-loss bugs. In Go codebases, the model traced concurrent access patterns across goroutines to identify real race conditions: not just "this looks like it might have a race" but "goroutine A writes to `cache.entries` on line 137 while goroutine B reads it on line 140 with no synchronization, which will panic under concurrent load." It named the specific data structure, the specific lines, and the specific failure mode. In TypeScript/React code, it followed event handler lifecycles to spot state-management bugs. It tracked how a `useEffect` cleanup function interacted with an async fetch, identified the exact window where a stale closure could cause a state update on an unmounted component, and proposed a cancellation-token pattern as the fix. In Ruby on Rails controllers, it identified authentication bypass vectors that arise from parameter handling edge cases, the kind of subtle permissiveness that a human reviewer might miss on a Friday afternoon but an attacker won't miss on a Saturday. In Java (Keycloak specifically), it caught contract mismatches between service interfaces and their implementations, tracing through multiple layers of abstraction to identify where a runtime exception would surface. In Python (Sentry), it identified silent failure paths where exceptions were caught too broadly, causing data-processing pipelines to swallow errors and produce incomplete results without any visible alert. ### **Cross-file reasoning** One of the most impressive capabilities, and the one that benefits most from the expanded context window, is the model's ability to connect findings across files. Given a diff, it traces helper-level contracts to downstream breakage and compares behavior across related methods, handlers, or providers. Opus 4.7 can tell you that the parameter *was* used by two downstream callers that the PR author forgot to update and that one of those callers will now silently fall back to a default value that breaks the billing calculation for enterprise accounts. Our analysis confirmed this pattern: the model "often connects helper-level contracts to downstream breakage and compares behavior across related methods, handlers, or providers." We observed this consistently across dozens of review sessions spanning five different language ecosystems. ### **Patch-oriented output** The review style is extremely code-centric, and this is where the practical developer experience shines: * **99.1%** of comments contain inline code references (specific variable names, function calls, line numbers) * **74.5%** include full code blocks demonstrating the issue or the fix * **78.0%** include actual diffs showing the proposed remediation ![Breakdown of review comment content showing 99.1% in-code references and 78% proposed remediation.](/content/assets/figure3-coments.png) In practice, most comments arrive with a ready-to-apply fix. The average comment runs **1,124 characters across 21 lines**, reading like a mini design review rather than a drive-by annotation. A typical comment opens with a bold, verdict-style summary ("Race condition in cache invalidation"), follows with a concise mechanism/impact explanation (2-3 paragraphs tracing the specific code path), and closes with a concrete diff wrapped in a collapsible `

` block. ## **The tone shift: Direct and opinionated** If you've used earlier Claude models for code review, the tone of Opus 4.7 will feel noticeably different. Anthropic describes it as "more direct and opinionated, with less validation-forward phrasing." Our evaluation quantifies this shift. Opus 4.7's review comments have an **assertiveness rate of 77.6%** and a **hedging rate of just 16.5%**. It leads with a bold, verdict-style summary of the issue, follows with a concise mechanism/impact explanation, and then presents a concrete patch. The language uses clear imperatives: "Guard against nil," "Prevent concurrent access," "Validate input before processing" rather than tentative suggestions. Our tone analysis summarized it well: "Comments read like detailed mini code reviews. They open with a bold, verdict-style summary of the issue, follow with 1–3 explanatory paragraphs, and then present a concrete patch in diff form. The tone is confident and directive, using clear imperatives rather than tentative phrasing." For maintainers, this is a welcome shift. When a model tells you "this will panic on nil input" instead of "you might want to consider checking for nil," you save cognitive overhead and can act on the feedback faster. In a busy review queue, that directness multiplies across dozens of comments per day. The hedging that does remain is well-placed. It appears primarily around subjective or domain-specific decisions, for instance, flagging a localization string as potentially incorrect and suggesting "please have a native speaker confirm." That's appropriate humility. The model is confident where it has evidence and careful where it doesn't. **Want to see this in action?** [Try CodeRabbit on your next PR](https://coderabbit.link/NcmblIe) \- free to start, no credit card required. See Opus 4.7-powered reviews on your own code. ## **What it's actually like to code with Opus 4.7** Benchmarks tell you how a model performs on a rubric. They don't tell you what it feels like to sit down with it and build something. Our engineering team has been hands-on with Opus 4.7 for coding tasks beyond code review, and a few patterns emerged. ### **It talks to you: A lot** The first thing you notice is how communicative the model is. As it works, the model narrates: what it's doing, why, which variables it's modifying, which files it's touching, and what its reasoning is at each step. The tone isn't conversational,it’s tactical. Every token carries information, optimized for context transfer rather than warmth. If you're new to working with AI coding assistants, this is great. You get a running commentary that doubles as a learning tool. But if you're an experienced developer who's used to terse, get-it-done interactions, it can feel over-communicative. There's a calibration period where you learn to skim the explanations and focus on the code output. The same depth we measured in the review benchmarks carries over. ### **Speed and reasoning scale together** Opus 4.7 has a strong sense of task complexity. When you give it something simple (rename a variable, add a guard clause, write a utility function), it moves fast. When you give it something genuinely hard (refactor a state machine, redesign an authentication flow, untangle a circular dependency), it takes more time to reason, and you can feel the difference. Even on complex tasks, the overall velocity is noticeably faster than previous models. The model seems to understand *how much thinking a task deserves* and allocates accordingly, so it doesn't waste your time over-reasoning on trivial work. In practice, this means you can move through a task backlog at speeds we haven't seen before. Simple changes fly by. Complex changes take longer but arrive with fewer bugs and better structure. ### **Code quality is high out of the gate** Across our first batch of hands-on sessions, the code quality was consistently strong. We encountered very few bugs during initial exploration, the kind of "it runs but doesn't work" failures that typically plague first-pass AI-generated code were notably rare. The model seems to get the logic right on the first try more often than not. There's a nuance here for frontend work. Opus 4.7 is excellent at the *logic* of UX: the placement of elements, the flow between states, the interactive behavior of components. But it doesn't have a great design taste. The UI it generates is functional and well-structured, but it won't win any design awards. If you're building a prototype or an internal tool, that's fine. If you're building a consumer-facing product, expect to bring your own design system and use the model for the logic layer. ### **It understands messy prompts** One thing that surprised us: Opus 4.7 is remarkably good at interpreting imprecise prompts. You don't need to write perfectly structured instructions. You can be vague, incomplete, or even somewhat contradictory in your prompt, and the model will generally infer what you actually meant and produce something useful. In real-world usage, developers are thinking faster than they're typing. They don't want to spend time crafting the perfect prompt, and with Opus 4.7, they don't have to. This tracks with what our benchmarks show in the code-review context. The model appears to reason about broader intent and context rather than treating each instruction as an isolated directive. ### **The self-review loop: Powerful but sometimes overeager** One of the more interesting behaviors we observed is that Opus 4.7 will often go back and review its own work after completing a task. It'll generate the code, then scan it for issues, then attempt to fix what it found, all without being asked. This self-correction loop can be genuinely valuable. It catches things the model missed on the first pass and improves the final output. But there's a downside. Sometimes the model overthinks it. It'll identify a "problem" in otherwise clean code and start reworking sections that didn't need to be touched, introducing unnecessary changes or even new issues in the process. The model's thoroughness occasionally tips over into over-correction. For developers, the practical advice is to review the model's self-edits with the same scrutiny you'd apply to any code change, and don't hesitate to roll back the second pass if the first one was already correct. ### **Surprising creative range** This was unexpected: Opus 4.7 is genuinely good at creative work. When we asked for titles, taglines, naming suggestions, and creative copy, the model produced results that felt original. It also performed well on graphical tasks: generating images, logos, vector graphics, and pixel art with a level of quality and coherence that went beyond what we expected from a model primarily known for code and reasoning. For developers who wear multiple hats (and most of us do), that creative range means you can use the same model for both the code and the marketing page that explains it. ## **Where we see room for improvement with Claude Opus 4.7** No model is perfect, and we'd rather be upfront about the rough edges than have you discover them yourself. 1. **Severity calibration is aggressive.** As the breakdown above shows, the model skews toward `critical` and `major`. While many of those labels are justified, the model also applies `critical` to speculative security surfaces, migration risks, and test-only failures that don't meet a strict rubric for that level. Identical comment text occasionally receives different severity labels across similar contexts, reflecting annotation instability we need to smooth out. We're tuning our post-processing pipeline to normalize these before they reach developers. 2. **Comment density is high.** The raw output is more "exhaustive audit" than "focused review." Not every PR needs 19 comments. Our filtering, ranking, and deduplication layers are essential to turning this into a usable signal that doesn't overwhelm developers. 3. **Duplicate findings across evaluation contexts.** We observed that the model sometimes produces near-identical comments across related code paths: for example, the same null-check warning applied to three similar handler functions. While each instance is technically correct, the repetition inflates apparent coverage and adds noise. Deduplication by normalized text \+ file/line is a necessary post-processing step, and we've seen cases where 30 \- 40 raw comments collapse to 10 \- 20 unique findings after deduplication. 4. **The over-correction instinct.** As we noted in our hands-on section, the model's self-review behavior (which is a strength in many contexts) can sometimes lead to unnecessary rework. In a code-review context, this manifests as the model flagging code patterns that are intentional or idiomatic as potential issues. The model's thoroughness is a feature, but its calibration on when to stop is still a work in progress. ![Graphic showing Opus 4.7 coding strengths like bug catching and review quality, with caveats.](/content/assets/figure4-pro-con.png) ## **What integrating Opus 4.7 means for CodeRabbit users** We're actively integrating Opus 4.7 into our review pipeline. Here's what you can expect as we roll it out: * **More bugs caught before merge.** The pass-rate and full-system improvements we detailed above translate directly into fewer escaped bugs. Over weeks and months, that compounds into meaningfully fewer production incidents, fewer hotfixes, and fewer late-night on-call pages. * **Feedback you can act on immediately.** Most findings arrive with inline code and ready-to-apply diffs. For many of them, you'll be able to apply the suggested change directly, review it, and move on, saving minutes per comment and hours per week. * **Better cross-file awareness.** If your PR updates a shared utility but forgets to update one of its three callers, Opus 4.7 is significantly more likely to catch that than previous models. Complex refactors and multi-file changes get smarter coverage. Opus 4.7 represents a step function in what's possible with AI-assisted code review. Stronger reasoning, broader context, more actionable output, configurable depth. The gap between AI review and expert human review continues to narrow. The AI isn't replacing the human reviewer. It's covering the ground that humans don't have time for. If you haven't tried CodeRabbit yet, there's never been a better time. Connect your repository in under two minutes. The model got a lot smarter, and so did your code reviews. [**Get started with CodeRabbit**](https://coderabbit.link/NcmblIe) \- connect your repo, get your first AI review in minutes. Free to try, no credit card required.

Introducing the CodeRabbit plugin for Codex

Juan Pablo Flores — Wed, 15 Apr 2026 00:00:00 GMT

Staying in flow matters. The moment a developer has to leave the current session to run a review somewhere else, wait for results, and context-switch back to act on findings, that momentum breaks. The CodeRabbit plugin for Codex brings AI-powered code reviews directly into the same surfaces where developers are already writing and iterating on code, so they can get structured feedback while the work is still fresh and move faster from draft to pull request. That matters most when a developer is already using agents to write, edit, and refine code inside the same session. Feedback lands better when it shows up before a change reaches a pull request, while the developer still has full context. Catching issues at that stage shortens review cycles, reduces back-and-forth, and helps teams ship with fewer surprises. %[https://youtu.be/NDcFLXQ3BhA] Bringing review into the same session where code is being written is the gap we wanted to close. A developer can ask Codex to "review my current changes with CodeRabbit" and the agent handles the rest: checking whether the [CodeRabbit CLI](https://docs.coderabbit.ai/cli) is installed, running authentication, executing the review against the working branch, and returning findings ordered by severity. All without leaving the terminal or the Codex app. ## How it Works At its core, the CodeRabbit plugin helps Codex run reviews without pushing the developer into a separate workflow. In practice, the flow looks like this: 1. A developer asks Codex to review the current changes using CodeRabbit. 2. The plugin checks whether the [CodeRabbit CLI](https://docs.coderabbit.ai/cli) is installed and whether the user is signed in. 3. Codex picks the right review target and runs CodeRabbit against the relevant changes. 4. Findings come back ordered by severity inside the same working session. 5. The developer can act on that feedback immediately or ask the agent to help fix the issues and review again. That first run experience is a big part of the value. On setup, the plugin guides the agent to check for required tooling, install it when needed, and only ask for user input when account sign in is necessary. The result is that review can stay closer to the work itself, which makes it easier for developers to act on findings immediately and for teams to move through review with less friction. ## Prerequisites Before installing the plugin, make sure you have:

**CodeRabbit CLI installed.** Install it globally by running:
```
curl -fsSL https://cli.coderabbit.ai/install.sh | sh
```
Restart your shell after installation to make sure the CLI is available.
**A CodeRabbit account.** You need to be authenticated before running reviews. You can sign in or [get started](https://app.coderabbit.ai/login?free-trial=) by running the authentication command within Codex and completing sign-in in the browser when prompted. The [CodeRabbit CLI](https://docs.coderabbit.ai/cli) is free to use, with [rate limits](https://docs.coderabbit.ai/cli#pricing-and-capabilities) on the free tier.

## Installation Codex supports plugin installation from the Plugin Directory in both the app and the CLI. The steps differ slightly depending on where you are working. ### In the Codex app: 1. Open *Plugins* from the sidebar. 2. Search for CodeRabbit and select Add to Codex. 3. If prompted, complete authentication during setup. ### In the Codex CLI: 1. Run `codex` and open `/plugins`. 2. Search for CodeRabbit and select Install plugin. 3. If prompted, complete authentication during setup. ### After installation: 1. Start a new thread and ask Codex to review your current changes. You can also type `@coderabbit` to invoke the plugin directly. 2. If you have not yet authenticated, the plugin will guide Codex through the sign-in process on first use. You will know the installation worked when Codex can recognize the review request, route it through CodeRabbit, and return findings inside the same session. The workflow works from both the Codex app and the CLI, so you can run reviews wherever you are already working. ## What Comes Next This first version is a starting point. We plan to keep refining how review feedback flows back into the agent and to expand the plugin with new skills as the workflows mature. Because users get updates through the plugin itself, new capabilities arrive without any extra setup. If you try it, we would love to hear how it goes. Tell us what to improve next in the [CodeRabbit subreddit](https://www.reddit.com/r/coderabbit/) or the [CodeRabbit Discord](https://discord.com/invite/coderabbit).

Our settings page was overwhelming our users. Here's what we did to fix it

Sahana Vijaya Prasad — Fri, 10 Apr 2026 00:00:00 GMT

CodeRabbit reviews code across every language, framework, and team size. Solo developers shipping side projects. Enterprises enforcing compliance across hundreds of repositories. That range means a lot of configuration knobs. More than a 2004 Honda Civic dashboard, certainly. Every setting exists because someone needed it. Path-specific review instructions, tool-level toggles, draft PR handling, labeling rules, tone control. Each one is a dealbreaker for somebody. Remove it and you lose them. Keep it and the settings page grows. Over time, our settings page became a wall of options that overwhelmed a lot of users — especially those new to CodeRabbit. Here's how we solved it and what we learned along the way. ## Who's actually in the settings page? Before changing anything, we looked at who was using the settings and what they were doing there. Four patterns showed up. 1. **The "just make it work" user.** They install CodeRabbit, maybe change the review language, and never come back. They want sensible defaults and the confidence that they're not missing anything important. 2. **The hands-on developer.** They spend fifteen minutes tuning things: adjusting the tone, setting up path-specific instructions, toggling walkthrough sections. Then they leave it alone. 3. **The platform team.** Enterprises with compliance requirements and standards enforced across every repo. They need access to everything because their use cases are the edge cases. 4. **The "config as code" user.** They don't want a UI at all. They want a YAML file in version control, reviewed in a PR, applied automatically. Configuration is infrastructure. A challenge for our team was recognizing that these weren't fixed user segments. A single user might belong to the "just works" group on Monday and the "config as code" group on Friday. ## **Why we couldn't just add an "Advanced" toggle** The obvious fix for settings overload is a simple view with an "Advanced settings" toggle. We tried it, but it didn't work for a couple of reasons. Our settings UI is generated directly from a JSON schema — the same schema that powers every `.coderabbit.yaml` file across millions of repositories. When a developer adds a property to the schema, the UI renders it automatically, without any additional frontend work. We didn't want to lose that. Bolting "basic" and "advanced" flags onto the schema would have added complexity to the one system we'd kept simple. But the deeper problem was simpler than architecture. We couldn't agree on what was basic versus advanced. Is "Auto-assign reviewers" a basic setting or an advanced one? For a solo developer it's a niche toggle; for an enterprise team it's a critical workflow. The basic/advanced split assumes users fall on a single axis of sophistication. Ours don't. A setting that's advanced for one persona is table stakes for another. ## Three views, one truth Since we couldn't restructure the schema or agree on two tiers, we tried something else: decoupling how settings are presented from how they're stored. We built three views over the same configuration. 1. **Concise** is the default. It shows a curated subset — the settings the vast majority of users actually change. We picked them based on usage data, support signals, and direct user feedback. The grouping is different from the schema too; "Behavior" pulls settings from multiple schema sections because that's how users think about them. 2. **All settings** show everything. Same auto-generated UI from the schema, same grouping as the YAML structure. The platform team that needs tool-specific rules or knowledge base settings finds it all here. 3. **YAML editor** is a Monaco editor with schema validation. Edit configuration as code, in the browser, with real-time error checking. All three read from and write to the same data. This shifted the conversation from "is this basic or advanced?" to "do most users need to see this?" without needing to maintain a separate schema. ## Making settings tangible Three views solved the structure problem, but there was a deeper issue: settings are abstract. "Enable sequence diagrams" means nothing if you don't know what a sequence diagram looks like in a PR review. So we added a live preview panel. It shows a mock PR review comment right next to the settings. ![Comparison of PR author-generated summary and AI-generated summary by CodeRabbit.](/content/assets/image4-5.png) ![GitHub pull request page displaying a sequence diagram, file changes, and estimated review effort.](/content/assets/image3-7.png) Toggle "sequence diagrams" off and the diagram disappears from the preview. Settings tagged with "Preview" tell the user which ones have a visible effect — instead of configuring blindly, users can see the result before they save. ![Dark UI settings panel with multiple options and active orange toggle switches.](/content/assets/image2-11.png) This solved one of the bigger problems that reorganizing the settings alone couldn't, which is users didn't understand what the settings controlled. ## **What we learned** Settings pages aren't glamorous. But get them wrong and every power user you have will feel it. Here are the three things that stuck with our team after this process and how we'll take it into our next redesign. 1. **There's no single right view for all users.** A solo developer and an enterprise platform team need fundamentally different things from the same settings page. What worked was giving them different paths to the same configuration and making transitions between them seamless. Someone who starts in Concise and can't find what they need should land in All Settings smoothly — not hit a dead end. 2. **Structure alone doesn't solve confusion.** We spent a lot of time on organization and grouping, but the preview panel ended up being one of the most impactful changes, and it had nothing to do with structure. Showing the output next to the input made settings concrete in a way that descriptions never could. 3. **Protect what already works.** We could have restructured the schema to make the UI cleaner. It would have broken every `.coderabbit.yaml` file out there. Keeping the data layer stable and putting the flexibility in the presentation layer let us ship a completely different experience without introducing regressions elsewhere. Ready to try it yourself? Start your [free trial of CodeRabbit](https://coderabbit.link/NcmblIe).

Gemini 3.1 Pro for code-related tasks: more focus, higher signal-to-noise

Erfan Al-Hossami — Thu, 12 Mar 2026 00:00:00 GMT

In practice, developers experience AI code review through the comments it leaves on pull requests: how often it finds real issues, how much noise it produces, and how actionable its feedback is. To answer those questions, we ran a benchmark comparing **Google’s Gemini 3.1 Pro** against our internal review baseline, a proprietary blend of OpenAI and Anthropic models tuned for CodeRabbit’s agentic PR review workflow. Using real pull requests with injected bugs, we measured not just detection rates but the structure and quality of the review comments themselves. The result reveals a clear trade-off: **Gemini leaves fewer, more focused comments with a higher signal-to-noise ratio, but it also surfaces fewer bugs overall.** ## **Methodology: How we benchmarked Gemini 3.1 Pro** ![Gemini 3.1 Pro for code-related tasks: More focus, higher signal-to-noise](/content/assets/gemini-3-1-pro-for-code-related-tasks-more-focus-higher-signal-to-noise-inline-image-2.png) Our benchmark uses an internal dataset composed of real GitHub pull requests into which specific, known error patterns must be addressed. Each error pattern (EP) has a ground-truth description of an issue. A model "passes" an EP if at least one of its review comments directly addresses or surfaces the root cause of the injected bug, either by proposing a concrete fix or by explicitly identifying the risk with an actionable direction. We used a suite of 25 hard PRs, each seeded with a known error pattern (EP). Our scoring focuses on: * Actionable comments only: Comments that get posted (not additional suggestions or outside-diff notes). * EP PASS (per comment): The comment directly fixes or surfaces the EP. * Important comments: Either EP PASS or another major/critical real bug. * Precision: EP PASS ÷ total comments. * SNR: Important ÷ (total − Important). We compared: * Gemini 3.1 Pro * CodeRabbit Production (a proprietary blend of OpenAI and Anthropic models tuned for CodeRabbit’s agentic PR review workflow) ## **Performance results** ### **Coverage and precision** ![Gemini 3.1 Pro for code-related tasks: More focus, higher signal-to-noise](/content/assets/gemini-3-1-pro-for-code-related-tasks-more-focus-higher-signal-to-noise-inline-image.png) Gemini 3.1 Pro trails on coverage by 4.3 percentage points. It generates 24% fewer actionable comments while landing a slightly higher proportion of them on target (33.3% vs 29.8%). On raw coverage, Baseline has the edge. The baseline’s nitpick-level comments detect **2 additional EPs** (+8.7pp) beyond its main comments, meanwhile Gemini does not detect EPs in its nitpick comments. ### **Signal quality** ![Gemini 3.1 Pro for code-related tasks: More focus, higher signal-to-noise](/content/assets/gemini-3-1-pro-for-code-related-tasks-more-focus-higher-signal-to-noise-asset.png) Gemini has a higher important-comment rate (77.8% vs 71.9%) and a better SNR (3.5 vs 2.6). Its comments are more likely to be classified as serious issues. It generates proportionally fewer minor comments than our baseline. On signal quality per comment, Gemini is ahead. ## **The behavioral layer** Most benchmark posts stop at pass rate and precision. This one doesn't. We ran tone classification on every comment to measure how each model communicates and found a meaningful difference. ### **Tone Profile** ![Gemini 3.1 Pro for code-related tasks: More focus, higher signal-to-noise](/content/assets/gemini-3-1-pro-for-code-related-tasks-more-focus-higher-signal-to-noise-asset-2.png) Gemini hedges more (0.229 vs 0.175) but is simultaneously more assertive (0.756 vs 0.703) and more confident (0.947 vs 0.939). This isn't contradictory; it reflects a style where Gemini softens its framing ("you might want to consider…") while its technical conclusions remain decisive. Its comments are longer on average but less likely to include code blocks or diff patches compared to Baseline. ### **The sharpest behavioral finding: Gemini knows when it's right** When we split tone metrics by pass/fail outcome, a strong pattern emerges: ![Gemini 3.1 Pro for code-related tasks: More focus, higher signal-to-noise](/content/assets/gemini-3-1-pro-for-code-related-tasks-more-focus-higher-signal-to-noise-asset-3.png) **Gemini's passing comments are 38% more assertive and 33% longer than its failing ones.** When Gemini catches a bug, it's measurably more decisive, more detailed, and more code-inclusive. Its internal confidence signal is reliable: if Gemini is assertive and long, it's probably right. Baseline shows the same directional pattern but the gap is narrower; its passing and failing comments look more similar to each other. Baseline's code block rate is nearly identical whether the comment passes or fails (88.2% vs 87.5%). Baseline applies effort broadly; Gemini concentrates it. This has a practical implication for teams using these models: **Gemini's comment tone is a useful proxy for comment quality.** A terse, hedged Gemini comment warrants more skepticism than an assertive, code-heavy one. Baseline's comments are more uniformly formatted regardless of accuracy. ## **Where Gemini shines** ![Gemini 3.1 Pro for code-related tasks: More focus, higher signal-to-noise](/content/assets/gemini-3-1-pro-for-code-related-tasks-more-focus-higher-signal-to-noise-asset-4.png) **Comment density when on target**: On the EPs where both models pass, Gemini's passing comments tend to be more specific. Its average passing comment is 1174 characters, nearly 32% longer than a typical Baseline passing comment (891 chars), and concentrates more on the root cause rather than symptom. ## **Where Gemini falls short** ![Gemini 3.1 Pro for code-related tasks: More focus, higher signal-to-noise](/content/assets/gemini-3-1-pro-for-code-related-tasks-more-focus-higher-signal-to-noise-asset-5.png) **Concurrency and threading (56% vs 78% on 9 EPs)**: This is the critical gap. Nine error patterns covered concurrency bugs, lock misuse, timing dependencies, race conditions, livelock. Gemini detected 5; Baseline detected 7. The 22-point gap on the dominant category in this dataset is what drives the coverage difference. ## **Conclusion & Limitations** Gemini 3.1 Pro produces higher-quality, more focused comments with better signal-to-noise, but it covers fewer bugs overall. Its SNR of 3.5 vs 2.6 means a developer reading Gemini's review is less likely to waste time on a low-quality comment. But with 60.9% EP detection vs 65.2% for Baseline, you're leaving more real bugs undetected. For codebases where concurrency bugs are a material risk, that gap matters. One finding worth tracking across future evaluations: **Gemini's internal tone calibration is strong.** Its assertiveness score seems to provide a signal as to whether a comment is likely to address the underlying issue. That said, these findings are scoped. The benchmark covers 25 error patterns across five repositories spanning Python, TypeScript, C/C++, and a mixed-language GitHub Actions codebase, but the error distribution is weighted heavily toward concurrency bugs (9 of 25 EPs), which is both where Gemini struggles most and where the gap is widest. Results may look different on codebases where OOP, transaction-semantic, or other bugs dominate. The tone calibration finding in particular should be validated on a broader error distribution before being trusted as a source of greater likelihood the comment is right. * * * *Evaluation conducted February 24, 2026. Baseline: Internal baseline on 25 difficult PRs evaluated for Gemini 3.1 Pro. Tone classification by GPT-5.1. Pass/fail determined by independent LLM judge per comment against ground-truth error description.* ***Interested in trying CodeRabbit? Get a*** [***14-day free trial!***](https://coderabbit.link/0XvQhxZ)

Faster AI code reviews with NVIDIA Nemotron 3 Super

Sahil Mohan Bansal — Wed, 11 Mar 2026 00:00:00 GMT

TL;DR: NVIDIA Nemotron 3 Super delivers high accuracy and faster throughput in CodeRabbit's self-hosted AI code reviews. We are happy to share that CodeRabbit is expanding its support for the [NVIDIA Nemotron](https://research.nvidia.com/labs/nemotron/Nemotron-3/) family of open models, upgrading from Nemotron 3 Nano to **Nemotron 3 Super** for the context gathering and summarization stage of our AI code review workflow. This upgrade is available for CodeRabbit's self-hosted customers running our container image on their own infrastructure. Nemotron 3 Super is used to power the context gathering and summarization stage before the frontier models from OpenAI and Anthropic are used for deep reasoning and generating review comments for bug fixes. With Nemotron Super, that review foundation just got significantly more capable. ## Upgrading from Nano to Super: Faster context gathering at scale We tested Nemotron 3 Super as a follow-up to our initial [support of Nemotron 3 Nano](https://www.coderabbit.ai/blog/coderabbit-ai-code-reviews-now-support-nvidia-nemotron), where we reported that a blend of open and frontier models allows us to improve the overall speed of context gathering and cost efficiency by routing different parts of the review workflow to the appropriate model family especially in the PR Summarization phase of code reviews. Nemotron 3 Super's larger context window and ability to run multi-token prediction (MTP) made it well-suited for the token-hungry task of context summarization. As our code review workflows grow more agentic and complex, we've run into two constraints that Nemotron 3 Super helps to address. **Context explosion:** Multi-agent workflows generate significantly more tokens than standard interactions because each step requires context from tool outputs, intermediate reasoning, repo signals, and more. Over the course of a long review, this volume of context increases cost and risks goal drift. **Thinking tax:** Complex agentic tasks require reasoning at every step, but routing every sub-task to a large frontier model makes the pipeline slow and expensive. The ideal solution is a mix of models where the reasoning model aligns with the type of task without escalating to the heaviest model available. ![Faster AI code reviews with NVIDIA Nemotron 3 Super](/content/assets/faster-code-reviews-with-nemotron-3-super-inline-image.png) *CodeRabbit architecture: using Nemotron Super for context gathering & summarization* This context building stage is the workhorse of the overall AI code review process and it is run several times iteratively throughout the review workflow. [NVIDIA Nemotron 3 Super](https://developer.nvidia.com/blog/introducing-nemotron-3-super-an-open-hybrid-mamba-transformer-moe-for-agentic-reasoning/) helps us with high-efficiency tasks and its large context window (1 million tokens) along with fast speed helps to gather a lot of data and run several iterations of context summarization and retrieval. Running these iterations many times iteratively throughout the code review cycle helps to enhance the review quality and lower the signal-to-noise ratio. ## How Nemotron 3 Super helps with review summaries When you open a Pull Request (PR), CodeRabbit’s code review workflow is triggered starting with an isolated and secure sandbox environment where CodeRabbit analyzes code from a clone of the repo. In parallel, CodeRabbit pulls in context signals from several sources: * Code and PR index * Linter / Static App Security Tests (SAST) * Code graph * Coding agent rules files * Custom review rules and Learnings * Issue details (Plan details, Jira / Linear / GitHub tickets) * Public MCP servers * Web search A lot of this context, along with the code diff being analyzed, is used to generate a PR Summary before any review comments are generated. ![Faster AI code reviews with NVIDIA Nemotron 3 Super](/content/assets/faster-code-reviews-with-nemotron-3-super-asset.png) *PR Summary generated by CodeRabbit, powered by Nemotron 3 Super* Summarization is at the heart of every code review and is the key to delivering high signal-to-noise in the review comments. Nemotron 3 Super is a 120-billion-parameter open model with 12 billion active parameters at inference. Its hybrid Mixture-of-Experts (MoE) architecture with transformer layers handling the reasoning and Mamba layers handling the high-volume, repetitive work of context processing during review summarization, which is critical for our code reviews. Predicting multiple tokens simultaneously also resulted in meaningfully faster inference which speeds up review summarization. All other code review tasks flow downstream from summarization. The faster the review summarization, faster the overall code review. Nemotron Super delivers much faster performance than Nemotron 3 Nano. Nemotron 3 Super can also hold a large codebase context including context from external sources (jira tickets, logs, project requirement docs, etc.) without losing state across long tasks. ## What Super unlocks beyond Nano CodeRabbit now supports Nemotron 3 Super (initially for its self-hosted customers) for the context summarization part of the review workflow, while the frontier models from OpenAI and Anthropic focus on finding hidden bugs. For customers this means faster PR summarization, faster code reviews without compromising quality. We are also delighted to support the [announcement from NVIDIA today](https://nvidianews.nvidia.com/news/nvidia-expands-open-model-families-to-power-the-next-wave-of-agentic-physical-and-healthcare-ai) about the expansion of its Nemotron family of open models and are excited to work with the company to help accelerate AI coding adoption across every industry. [Get in touch](https://www.coderabbit.ai/contact-us/sales) with our team to access CodeRabbit’s container image if you would like to run AI code reviews on your self-hosted infrastructure.

Introducing one of the most requested CodeRabbit features: multi-repo analysis.

Konrad Sopala — Fri, 06 Mar 2026 00:00:00 GMT

If you've ever merged a pull request that passed every check, looked clean in review, and then broke a downstream service ten minutes later…you already know the problem. When your architecture spans multiple repos (microservices, shared libraries, separate frontend and backend packages) a change in one place can silently break things in another. A renamed field in your API response schema? Looks great in the PR, but the three services that parse that response have no idea what's coming. This is one of the most common pain points we hear from teams running multi-repo setups and it's been one of the most requested features among our customers. So we built it. ![Introducing one of the most requested CodeRabbit features: Multi-Repo](/content/assets/coderabbit-multi-repo-analysis-inline-image-2.png) ## Introducing Multi-Repo Analysis [Multi-Repo analysis](https://docs.coderabbit.ai/integrations/knowledge-base#linked-repositories-cross-repo-review) is a new CodeRabbit feature available to Pro, Pro Plus, and Enterprise tier users that let you connect related repositories so that CodeRabbit pulls context from across all of them during code reviews. Think microservices, shared libraries, API contract changes, or any setup where a change in one repo can quietly break another. More information on linked repositories per plan can be found [here](https://docs.coderabbit.ai/knowledge-base/multi-repo-analysis). When a pull request modifies a shared API, type definition, or database schema, CodeRabbit automatically explores your linked repositories for downstream impact. Instead of reviewing changes in isolation, you get the full picture before you merge. ![Introducing one of the most requested CodeRabbit features: Multi-Repo](/content/assets/coderabbit-multi-repo-analysis-inline-image.png) ## Setup & configuration Before you start using it, there are some [platform-specific requirements](https://docs.coderabbit.ai/integrations/knowledge-base#platform-requirements) to go through, to make sure the CodeRabbit bot has read access to all linked repositories:

Platform	Requirement
GitHub	The CodeRabbit GitHub App must be installed on all linked repositories. Inaccessible repositories are skipped, and a warning appears in the review summary.
GitLab	The bot token must have read access. Tokens are typically scoped to the group or instance.
Bitbucket Cloud	The bot token must have read access. Tokens are scoped to the workspace.
Azure DevOps	The PAT must have read access. Tokens are scoped to the organization.

Once that's configured, you can finish the setup in two ways: * **Through the CodeRabbit web interface** * Head to [Repositories](https://app.coderabbit.ai/settings/repositories) section and select a repository that you would like to add a linked repository to * Switch off the *Use Organization Settings* toggle * Go to the [Knowledge Base](https://app.coderabbit.ai/organization/settings?tab=knowledge_base) and find the Linked *Repositories* section. * Add a new linked repository with instructions that will guide CodeRabbit during review. ![Introducing one of the most requested CodeRabbit features: Multi-Repo](/content/assets/coderabbit-multi-repo-analysis-asset.png) * **Through YAML configuration** * Add a *linked\_repositories* section under *knowledge\_base* in your [.coderabbit.yaml](https://docs.coderabbit.ai/getting-started/yaml-configuration) file. ```plaintext # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json knowledge_base: linked_repositories: - repository: "myorg/backend-api" instructions: "Contains REST API endpoints and database models" ``` Keep in mind that as of now, each repository configuration supports only one linked repository, though multiple linked repos are planned in the future. For full setup details, check the [docs](https://docs.coderabbit.ai/knowledge-base/multi-repo-analysis). ## See it in action If you want a hands-on walkthrough, one of our Developer Advocates put together a tutorial that shows Multi-Repo analysis catching real cross-repo issues in a review. %[https://youtu.be/BrT4rSHhA10?si=FzPUzcBUgqXdHXmZ] ## Try it now A Multi-Repo analysis feature is available today. Connect your repos, open a PR, and let CodeRabbit show you what you've been missing. [Get started](https://app.coderabbit.ai/login)

A semantic history: How the term 'vibe coding' went from a tweet to prod

David Kravets — Thu, 05 Mar 2026 00:00:00 GMT

A year is an eternity in AI time. In February 2025, Andrej Karpathy dropped a [tweet-sized cultural marker](https://x.com/karpathy/status/1886192184808149383) into the software world: “vibe coding.” The phrase stuck because it captured a visceral shift in the developer experience. Instead of grinding through syntax, you describe an intent in plain English, watch the code manifest, and nudge the output until reality matches your vision. In this initial vision of vibe coding, Karpathy famously framed it as engineering where you, “fully give in to the vibes, embrace exponentials, and forget that the code even exists.” ![A semantic history: How the term 'vibe coding' went from a tweet to prod](/content/assets/a-semantic-history-how-the-term-vibe-coding-went-from-a-tweet-to-prod-inline-image.png) Karpathy’s point wasn't just that AI could help you code. We’ve had copilots for years. The point was that AI made it tempting to treat code as a steerable draft rather than something requiring line-by-line authorship. One year later, the "vibe" has evolved. Vibe coding doesn’t just refer to what we’re doing when coding a hobby project or slapping together a prototype anymore. We are managing an explosion of machine-generated logic destined for production systems, and we’re often calling that vibe coding, too. But is it really vibe coding? And if it’s not, what should we be calling it? And how should we be treating it in production systems? ## The semantic drift: From play to production ![A semantic history: How the term 'vibe coding' went from a tweet to prod](/content/assets/a-semantic-history-how-the-term-vibe-coding-went-from-a-tweet-to-prod-inline-image.jpg) When vibe coding was honored as a 2025 [Word of the Year by Collins Dictionary](https://blog.collinsdictionary.com/language-lovers/collins-word-of-the-year-2025-ai-meets-authenticity-as-society-shifts/), it signaled that the workflow had gone mainstream. But as the phrase broadened, so did the stakes. Vibe coding today has become shorthand for any prompt-driven development, rather than the specific prompting experience Karpathy originally described. This drift isn’t a linguistic accident. More than anything, it’s a signal that many engineering teams have added AI to their stacks and are struggling with the quality of its output and its downstream effects. In fact, the use of vibe coding to describe the generation of code for production-grade systems has taken on somewhat negative connotations. The term was invoked by many precisely to emphasize that, in some cases, relying too much on AI-generated code at work was trusting an LLM’s vibes in a way better suited for a weekend project, rather than for customer-facing applications of a publicly listed company. For example, as AI coding agents were adopted by more teams in 2025, devs on LinkedIn renamed themselves [Vibe Code Cleanup Specialists](https://www.linkedin.com/posts/deveshbhardwajj_ai-startups-engineering-activity-7373218403068788736-VXlT) in their profiles (a joke we ran with for our [AWS Re:invent booth](https://x.com/yoimkonrad/status/2013539187224375400?s=46&t=OXgZw0VC393gp6RVxqzq0A) this year). What’s more, Collins’ competitor Merriam Webster went in another direction for their word of the year choice. They chose [slop](https://www.merriam-webster.com/wordplay/word-of-the-year), highlighting the large gulf that still existed between AI optimism and AI output. Indeed, for many, the promise of AI coding agents have translated into hours spent reviewing AI slop, reworking code [due to unclear prompts or intent](https://www.coderabbit.ai/blog/the-hidden-cost-of-ai-coding-agents-isnt-from-ai-at-all?), and dealing with incidents or bugs in production. As we move from "throwaway weekend projects" (as Karpathy originally described his use of vibe coding) to core infrastructure, the constraints of software development have shifted. We no longer have a creation problem. We have a confidence problem. And that lack of confidence was reflected in how the word came to be used throughout 2025 for almost all uses of AI for coding, even those at work. ## A word under pressure: Why devs both love and hate vibe coding ![A semantic history: How the term 'vibe coding' went from a tweet to prod](/content/assets/a-semantic-history-how-the-term-vibe-coding-went-from-a-tweet-to-prod-asset.jpg) Some of the biggest tech companies brag about how a growing amount of their code, if not all, [is written by AI](https://www.coderabbit.ai/blog/ai-code-metrics-what-percentage-of-your-code-should-be-ai-generated). That can’t be accomplished by simply forcing devs to use AI. Many, if not most devs, appreciate the time and cognitive savings of [using AI agents](https://www.atlassian.com/blog/developer/developer-experience-report-2025) for certain tasks and types of code. But that’s not the full story. For many devs, AI coding agents often actually equal more work, and that’s measurable in how the most experienced people in the room are spending their hours. Fastly’s [recent survey](https://www.fastly.com/blog/senior-developers-ship-more-ai-code) of 791 professional developers illustrates this shift in real-time. The data shows that senior developers are far more likely to ship large amounts of AI-generated code. * **Volume:** Roughly one-third of senior developers report that about half their shipped code is now AI-generated (compared to just 13% of juniors). * **The review tax**: Nearly 30% of senior devs report that editing and auditing AI output offsets most of their initial time savings from AI. This tells us that the most experienced engineers aren't using AI to work less. They are using it to manufacture more complexity, which they then have to manage. That’s not just because they’re coding more. It’s also because reviewing AI code is simply harder. Our own [State of AI vs. Human Code Generation study](https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report), for example, found that AI generated code had 1.7x more bugs and issues and 1.4x more critical issues. Today, where it seems like everyone is vibe coding at work, code review is no longer a predictable ritual at the end of a sprint. Now, it is the primary mechanism that makes high-velocity engineering safe. Or, more so, it’s the area where issues and bugs now slip through more frequently and requires significantly more time and attention to get right. In a way, it must feel to some developers like a Faustian bargain. In return for manually writing less code, they’re having to manually review more of it. And those reviews have increasingly become stressful, load bearing stages of the [software development lifecycle](https://www.coderabbit.ai/guides/agentic-sdlc). But one glance at the surge of headlines about incidents clearly shows that the normal review cycle (and likely senior devs) are breaking under the added load. The difficulty of finding all the additional issues AI generates is leading to a wave of incidents that are suspected to be from or have been traced back to AI-generated code. ## Incidents changed the vibe coding conversation ![A semantic history: How the term 'vibe coding' went from a tweet to prod](/content/assets/a-semantic-history-how-the-term-vibe-coding-went-from-a-tweet-to-prod-asset.png) As AI-generated code moved from prototypes into production systems, incident data began to reflect the strain. Industry outage tracking in early 2025 showed a [measurable spike](https://www.coderabbit.ai/blog/why-2025-was-the-year-the-internet-kept-breaking-studies-show-increased-incidents-due-to-ai) in global service disruptions during peak AI adoption months, before stabilizing later in the year. That analysis raised a harder question: not whether AI can generate code, but whether teams are verifying it rigorously enough before it ships. At the same time, teams reported a growing review burden. Faster generation meant more logic to inspect, more edge cases to reason through, and more subtle regressions to catch. When verification practices don’t scale at the same rate as output, small defects are more likely to escape into production. Recent high-profile events reinforced the point. For example, news reports detailed an [AWS disruption](https://www.reuters.com/business/retail-consumer/amazons-cloud-unit-hit-by-least-two-outages-involving-ai-tools-ft-says-2026-02-20/) in which an internal AI coding assistant was involved in changes that contributed to a 13-hour interruption of a cost-management service. While misconfigured access controls, not autonomous AI behavior, were later attributed as the root cause, it was, ultimately, the AI who made the change that took down the system. Similarly, Moonwell recently dealt with an incident that saw it accidentally issue [$1.8 million in bad debt](http://finance.yahoo.com/news/oracle-error-leaves-defi-lender-082155372.html?guccounter=1&guce_referrer=aHR0cHM6Ly93d3cuZ29vZ2xlLmNvbS8&guce_referrer_sig=AQAAANRP-C1XZnEqcpitPfwnrBhu46h20KdgmG1GWVmifMnb9gbTRBImfb7BhTM9pFVB-NZJqUxpY3SiPnlPVl4vLKL1LS-uYTKLirpb5NNKQY-V04HTCX3e3jVsJxPSQ-ubBiMlnGWf4503odt6oZb0od3VawMW8LZZMdoAWGnMlzR_) after an incident that many are suggesting was caused by AI-powered development. Kimi chatbot experienced [reliability issues](https://datainnovation.org/2025/01/moonshot-ai-betting-big-on-long-context-confronting-the-challenges-of-scale-and-reliability/) and outages amid surging demand as the company scaled aggressively, highlighting how systems can strain when rapid AI adoption outpaces infrastructure maturity. This is where “vibe coding” shifted in tone further. What began as a playful description of prompt-driven creation took on an even more skeptical edge in production contexts. Not because engineers reject AI leverage, but because incidents make the verification gap visible. When change accelerates without proportional review rigor, risk increases and devs become frustrated. ![A semantic history: How the term 'vibe coding' went from a tweet to prod](/content/assets/a-semantic-history-how-the-term-vibe-coding-went-from-a-tweet-to-prod-asset-2.png) ## The rise of ‘agentic engineering’: Vibe maturity or the same slop by another name? Karpathy recently refined his vibe coding thesis, noting that while the vibe started as an exploration, it has matured into what he and others describe as “agentic engineering.” This seems like an attempt on Kaparthy’s part to decouple AI-driven development from the baggage that the term vibe coding carries. But it’s also meant to mark a different kind of workflow that requires a fundamental shift in how we define technical oversight. As Karpathy recently noted on X, the industry is moving toward a model where speed and rigor are no longer mutually exclusive. “Today (1 year later), programming via LLM agents is increasingly becoming a default workflow for professionals, except with more oversight and scrutiny. The goal is to claim the leverage from the use of agents but without any compromise on the quality of the software,” Karpathy recently [wrote](https://x.com/karpathy/status/2019137879310836075) on X. ![A semantic history: How the term 'vibe coding' went from a tweet to prod](/content/assets/a-semantic-history-how-the-term-vibe-coding-went-from-a-tweet-to-prod-asset-3.png) This new paradigm is meant as a distinction from the vibe coding era. The job is no longer to coach the model or to vibe with it but to plan for and verify its logic. Whether “agentic engineering” will catch on as the main way to describe AI-driven development or whether the derogatory use of vibe coding in a professional context will continue remains to be seen. ## What’s next for vibe coding? The future of the word and working style Here’s the optimistic reading of the last year. Vibe coding didn’t break engineering, even if it did increase incidents. Instead, it forced us to define what the craft actually is and imagine how it might evolve in the coming years. That’s led some to talk about how we’re on the cusp of a new [golden age of software engineering](https://newsletter.pragmaticengineer.com/p/the-third-golden-age-of-software), and others to proclaim that [developers are dead](https://www.coderabbit.ai/blog/developers-are-dead-long-live-developers). Whether autonomous agents will ever replace developers is still highly contested. But what’s crystal clear is that the future of vibe coding as a word is entirely dependent on solving the quality problem AI-driven development currently faces. That’s the only way that “vibe coding” is set to become less derogatory and have a chance of evolving into Karpathy’s preferred term for it: “agentic engineering.” But to get there, we’ll need to adopt and perform rigorous “vibe checks” as an industry (that’s our suggestion for the 2026 word of the year, if you’re listening Collins and Merriam Webster). In its original cultural context, a vibe check is a gut-check, a moment of scrutiny to see if something is as real as it claims. At CodeRabbit, we first used this term in May 2025 to describe how our AI code reviews can help ensure your code’s quality. In 2026, it’s our belief that rigorous vibe checks must become a technical standard. The same tipping points we saw around the adoption of agentic AI in 2025 must happen in 2026 around tools that are focused on verifying and testing AI generated code. These vibe checks can include a number of different safeguards and systems designed to keep AI slop and issues from interfering with your production system, including things like AI code review. A vibe check is, essentially, your quality gate, something AI code makes more critical than ever before. We might have fallen in love with vibe coding because it named a feeling, the thrill of code appearing faster than the brain can keep up. But going forward, we need less vibing and more checking. That’s the only way vibe coding grows up. ***Need a vibe check?*** [***Try CodeRabbit free today.***](https://coderabbit.link/Gg1HxjW)

CodeRabbit tops the first independent AI code review benchmark

Sahil Mohan Bansal — Tue, 03 Mar 2026 00:00:00 GMT

AI code review benchmarks have mostly been published by other code review vendors (whose tools always seem to come out on top in their benchmarks). We've [written before](https://www.coderabbit.ai/blog/framework-for-evaluating-ai-code-review-tools) about why we don’t think vendor generated benchmarks provide the credibility developers actually need when choosing an AI tool. So, we’re glad to see that someone has finally built the first independent benchmark covering nearly 300,000+ real-world PRs reviewed by CodeRabbit. Martian's [Code Review Bench](https://codereview.withmartian.com/) is the first independent public benchmark to evaluate AI code review tools using real developer behavior and CodeRabbit comes out on top. Their leaderboard shows **CodeRabbit has the highest recall** of any tool, almost 15% more than the next closest tool. In plain terms: **CodeRabbit finds more real bugs** than anyone else. CodeRabbit also tops the overall chart with the **highest F1 score** (balance of precision/recall), with **a 51.2% score**, more than any other code review tool. Precision refers to the accuracy of a tool. Recall refers to the comprehensiveness of a tool CodeRabbit balances both and delivers the most accurate AI code reviews. [![Code Review Benchmark by Martian](/content/assets/coderabbit-tops-martian-code-review-benchmark-inline-image.png)](https://codereview.withmartian.com/) *Code Review Bench results: y-axis shows the recall rate, x-axis shows the precision* ### What is Code Review Bench? [Code Review Bench](https://withmartian.com/post/code-review-bench-v0) is a new benchmark published by [Martian](https://withmartian.com), a research lab with team members from DeepMind, Anthropic, and Meta. They evaluated 10 tools across approximately hundreds of thousands of PRs. Their [methodology and code](https://github.com/withmartian/crb) are fully open source. What makes Code Review Bench different from previously published code review benchmarks is their two-pronged approach involving real-world data from developers and a base gold set of known bugs to look for. * **Online benchmark:** analyzes code review comments that developers actually accept or reject across open source repos. When a developer fixes an issue found by a code review tool, that's a signal that the review comment was useful. When they ignore it, that's valuable data, too. * **Offline benchmark:** runs every code review tool on the same 50 PRs and analyzes them against a curated set of previously identified bugs called the “gold set.” This is a controlled but influenced comparison where human annotation is required to classify a review comment as a real bug or a false positive. ### Code Review Bench Results In the online benchmark that’s grounded in real developer behavior, and the one Code Review Bench itself calls the headline metric, CodeRabbit ranks **#1 in F1 score** (harmonic mean of precision and recall) among all 10 tools included. [![CodeRabbit tops the first independent AI code review benchmark](/content/assets/coderabbit-tops-martian-code-review-benchmark-inline-image-2.png)](https://codereview.withmartian.com/) *Code Review Bench results analyzed from Jan-Feb 2026* Martian measured CodeRabbit's reviews **across nearly 300,000 pull requests** over a 2 month period, one of the largest sample in the dataset. That represents tens of thousands of developers, on real projects, deciding whether to act on what we flagged. Developers act on CodeRabbit's suggestions at a meaningful rate. Its 49.2% precision means roughly one in two comments leads to a code change. This, combined with a higher recall rate than any other tool, leads to the highest F1 score among all tools. ### Optimizing the precision-recall tradeoff But what really are Precision and Recall when it comes to code reviews? Put simply: Precision is a measure of the % of true positives out of all review comments from a tool. Mathematically, this is a measure of the true positives in a tool's comments. If a tool returned 100 comments and it had 80 true positives, and 20 false positives (wrongly identified as a bug) then its precision rate is 80%. ![CodeRabbit tops the first independent AI code review benchmark](/content/assets/coderabbit-tops-martian-code-review-benchmark-asset.png) Recall is a measure of the % of total true positives found by that tool out of all bugs that exist. Mathematically, this is a measure of the comprehensiveness in a tool's comments. If a tool returned 50 comments but there were actually 100 real bugs in the PR then its recall rate is 50%. ![CodeRabbit tops the first independent AI code review benchmark](/content/assets/coderabbit-tops-martian-code-review-benchmark-asset-2.png) There's a common assumption when comparing code review tools that the best code review tool is the one with highest precision. Fewer comments, higher precision, sounds intuitive right? However, CodeRabbit has always taken a slightly different approach. CodeRabbit is specifically engineered to have a good balance between precision and recall. We would rather flag a real bug you choose to dismiss than miss a real bug you needed to see. This comes from optimizing not just for high accuracy (precision) but for both high accuracy and surfacing more bugs (recall). Optimizing for both precision and recall leads to catching the highest number of real bugs. [![CodeRabbit tops the first independent AI code review benchmark](/content/assets/coderabbit-tops-martian-code-review-benchmark-asset-3.png)](https://codereview.withmartian.com/) *High precision, high recall, leads to more true positives being found.* The reason why the online benchmark approach is important is that the offline approach often will flag a false positive when a code review tool finds a real issue that the benchmark's gold set doesn't include. High-volume tools like CodeRabbit simply surface more issues that the gold set didn't anticipate leading to bias against tools with higher recall. By not relying on a curated set of "correct” answers, the online benchmark sidesteps this oversight. It counts what developers actually did in their PRs. They found that devs acted on CodeRabbit's comments at roughly the same rate as tools that comment half as often. By surfacing more comments CodeRabbit ultimately catches more critical bugs. ![CodeRabbit tops the first independent AI code review benchmark](/content/assets/coderabbit-tops-martian-code-review-benchmark-asset-4.png) ## Offline benchmark tells a different story (and that's the point) In the 50-PR offline comparison mentioned in [Code Review Bench](https://codereview.withmartian.com/), CodeRabbit showed a lower F1 score than the online analysis that included comments accepted by devs in real PRs. They are transparent that the offline benchmark has "significant divergence" from the online data that needs to be improved. They describe two specific problems that affect high-volume tools disproportionately: * **The gold set is incomplete.** The offline comparison started with a dataset of known bugs curated by two other code review vendors but they found some comments that were scored as "false positives" were actually real issues the gold set didn't include. * **Offline and online benchmarks are designed to disagree.** Code Review Bench built both specifically so they could identify these kinds of gaps in the gold set. As they expand the gold set and calibrates against real-world behavior, they expect the offline rankings to shift. ## Conclusion We are glad to see this independent confirmation that our approach works in the real-world by catching more bugs that other review tools miss. We've always believed that the job of a code review tool is to catch as many critical bugs as possible and trust developers to decide which ones matter. ![CodeRabbit tops the first independent AI code review benchmark](/content/assets/coderabbit-tops-martian-code-review-benchmark-asset-5.png) We also allow for configurability to tune the kinds of bugs caught based on each team’s needs and improve reviews over time. CodeRabbit is built to be thorough by default while also being configurable for your team’s noise tolerance with controls like Chill vs Assertive review profiles (fewer vs. more comments), path-based instructions, and Learnings to customize your reviews to the issues you want surfaced. ![CodeRabbit tops the first independent AI code review benchmark](/content/assets/coderabbit-tops-martian-code-review-benchmark-asset-6.png) Martian's Code Review Bench, grounded in real developer behavior, shows that our approach works and is the better choice for teams who want to ship fast (but not break things). We’ll continue to watch how the Code Review Bench results improve over time. In summary: nearly 300,000, PRs, 53.5% recall rate, and #1 in F1 metrics. Not on a curated lab test using a static list of known bugs but based on real-world developer signals. That's what CodeRabbit is built for. Explore the full results and methodology: * [Code Review Bench results](https://withmartian.com/post/code-review-bench-v0) * [Methodology and open source code](https://github.com/withmartian/crb) Interested in trying CodeRabbit? [Start your 14-day free trial](https://www.coderabbit.ai/) today.

AI is burning out the people who keep open source alive

Aleks Volochnev — Sat, 28 Feb 2026 00:00:00 GMT

Over the past few months, one refrain has been heard consistently in open source communities: “AI slop.” It shows up in [LinkedIn discussions from CEOs](https://www.linkedin.com/feed/update/urn:li:activity:7417250625391915009/) who rarely complain publicly. ![Coding agents are ripping apart OSS communities ](https://cloudmate-test.s3.us-east-1.amazonaws.com/uploads/covers/695d7ae2e2a2e9cdf5199232/1cea7f89-776b-4531-9194-23485f81892e.png) It shows up in [Reddit threads](https://www.reddit.com/r/opensource/comments/1q3f89b/open_source_is_being_ddosed_by_ai_slop_and_github/), where users only think AI slop has become worse over the past few months. ![GitHub isn't helping OSS projects fight AI slop](https://cloudmate-test.s3.us-east-1.amazonaws.com/uploads/covers/695d7ae2e2a2e9cdf5199232/98cfdc23-407c-4b02-8e17-e90da5cec839.png) In [Twitter posts from popular OSS projects](https://x.com/tldraw/status/2011911073834672138?s=20) who have begun automatically closing pull requests from external contributors. ![tldraw shut down PR from external contributors due to AI slop](https://cloudmate-test.s3.us-east-1.amazonaws.com/uploads/covers/695d7ae2e2a2e9cdf5199232/bcd78199-003a-4b72-bffe-2c7d148ae9b5.png) The theme is consistent: projects are being flooded with AI-generated pull requests that merge cleanly but don’t actually work. Some maintainers are adding stricter contribution templates. Others are asking for smaller PRs. A few have temporarily limited new contributors just to regain control of their review queues. This isn’t an anti-AI backlash. Most of the people raising these concerns use AI themselves. Even a prominent figure like Peter Steinberger, the creator of the popular AI agent [OpenClaw](https://openclaw.ai/) (which was recently acquired by OpenAI), is struggling with the overwhelming volume of PRs. Despite his own heavy reliance on AI tools, he has [publicly requested a better solution](https://x.com/steipete/status/2023057089346580828?s=46) to manage the pace of incoming contributions. ![OpenClaw drowning in AI Slop pull requests](https://cloudmate-test.s3.us-east-1.amazonaws.com/uploads/covers/695d7ae2e2a2e9cdf5199232/fcaae4a4-e587-4c63-916d-475474f07666.png) ## My own experience: Slop galore As a maintainer of open source projects, I integrated AI into my coding process. Initially, this combination felt ideal as development gained momentum, contributors submitted fixes faster, and issues were addressed more consistently. The shift in open-source development felt invigorated; problems that once took months were now resolved in hours. This speed was beneficial when AI augmented human judgment. However, as the focus moves toward fully automated agents, a crucial element is diminishing. The issue with many of these automated pull requests isn't technical syntax, but a failure of judgment. What I didn’t expect was how much time I would start spending reviewing pull requests. That wasn’t because contributors suddenly got worse, or because AI made people careless. Writing code stopped being the hard part. Open source has always depended on the hard parts to act as a natural filter on what gets proposed, reviewed, and ultimately merged. ## Open source PRs are coming faster, but maintainers are still human Surveys like [Stack Overflow’s Developer Survey](https://survey.stackoverflow.co/) show that a large majority of developers (80%) already use, or plan to use, AI coding tools. You don’t need a report to feel that shift, though, you can see it immediately in active repositories. Pull requests arrive faster, they tend to be larger on average, and they often carry the unmistakable shape of something generated from a single prompt or a short back-and-forth with a model. That isn’t a criticism, I use AI tools every day myself and I would be lying if I said I wanted to go back to the way things were before. This is simply what happens when contribution becomes cheap. The problem many open source maintainers are running into is that reviewing those contributions did not get the same speed boost and it has become more cognitively demanding. I have seen pull requests that introduce a brand-new abstraction layer to solve what was previously a five-line change, complete with interfaces, helpers, and configuration flags that technically work but dramatically increase complexity. Nothing is “wrong” in the obvious sense. The surface-level checks-passing tests and a green CI-suggest no obvious errors. However, the reviewer's burden increases significantly; they must now decipher not only what changed, but also the rationale behind the specific implementation chosen and whether it subtly contradicts long-standing architectural decisions. ## Why reviewing AI-generated code feels harder Reviewing AI-generated code is strange because it often looks correct at first glance. The formatting is clean, naming is reasonable, and the logic flows in a way that feels intentional. Yet, you find yourself slowing down, because something does not quite sit right. You start asking questions that are hard to answer quickly. * Why does this abstraction exist at all? * Why does this function do three things instead of one? * Why does an approach technically work but feel fragile under real-world conditions? A frequent issue I've observed is with error handling in AI-generated code. For instance, I've reviewed changes where the AI catches and logs every conceivable exception. While this appears robust on the surface, in reality, it can mask critical failures that other parts of the system rely on to be visible. The change functions correctly in isolation but introduces subtle flaws by violating system-wide assumptions. Catching that kind of issue requires knowing how downstream components expect failures to propagate, which is rarely written down. Those are not questions a linter answers, and they are not always questions a contributor can explain either, especially when they do not fully own the context behind the change. Most open source projects rely on a large amount of implicit knowledge that never makes it into documentation. There are old design decisions buried in issues from years ago, conventions enforced socially rather than technically, and trade-offs that only make sense if you remember why something was done a certain way in the first place. AI does not know any of that unless someone teaches it and contributors often do not know it either. That gap is where much of today’s review friction lives. ## The real cost lands on maintainers Research on open source sustainability has shown for years that a [tiny percentage of contributors do the majority of review](https://arxiv.org/html/2312.17236v1#:~:text=Figure%201%3A,area%20of%20B.) work. So, in open source, this kind of review almost always falls on the same small group of people. You see the same names over and over again on merged pull requests. Maintainer burnout did not start with AI, but AI made it more visible by removing many of the buffers that once kept it manageable. More pull requests come in. Fewer of them are obviously wrong. Each one requires more careful mental simulation to feel confident it will not cause problems later. That often means scrolling through files, replaying edge cases in your head, and asking yourself what you are about to miss this time. It is slow, quiet work, and does not scale well. As the effort required for review outpaces the availability of reviewers, projects suffer a gradual decline rather than an abrupt failure. The symptoms include slower responses and accumulating unresolved issues. This eventually discourages contributors, who cease participation due to the lack of timely feedback. What is interesting is that many projects have started adapting without making a big announcement about it. You see stricter automated checks appear, clearer expectations for pull request descriptions, and more effort spent validating changes earlier in the process, before a maintainer ever looks at them. Reviews are moving earlier in the workflow not because maintainers want to be gatekeepers, but because human attention is limited and expensive. It makes more sense to spend it on intent and design than on catching avoidable mistakes. ## AI didn’t break open source, it exposed the bottleneck AI didn’t break open source. It exposed what was already straining under the surface: [reading code is harder than writing it](https://www.coderabbit.ai/blog/its-harder-to-read-code-than-to-write-it-especially-when-ai-writes-it#https://www.coderabbit.ai/blog/its-harder-to-read-code-than-to-write-it-especially-when-ai-writes-it?heading-reading-code-is-actually-harder-than-writing-it), especially when AI writes most of it. The real shift wasn’t just faster generation. It was volume. Suddenly, maintainers weren’t reviewing a handful of thoughtful pull requests each week. They were reviewing AI-generated diffs at a pace no human brain signed up for. Writing got easier. Reviewing got exponentially harder. That’s where [AI code reviews](https://www.coderabbit.ai/) started to make sense to me - not as a vague “AI assistant,” but as an always-on reviewer that runs before a human ever has to burn cognitive energy on a PR. When I started using CodeRabbit on my open source projects, it caught the obvious things immediately: off-by-one errors in pagination logic, missing error handling around external calls, subtle edge cases that only fail under specific configurations, and security issues that would have been painful to debug after a release. It flagged missing tests that technically “worked” but weren’t defensible. It forced contributors including myself to clean up code before asking for human reviewer time. Instead of spending my energy scanning for logic bugs or basic security issues, I could focus on intent. * Does this align with the project’s direction? * Does this abstraction make sense long term? * Is this the right tradeoff? The AI handled the mechanical correctness checks so humans could stay strategic. In my experience, automating feedback early in the pull request process speeds things up and reduces review cycles. When contributors know their PR will get AI code review feedback instantly, they submit smaller changes, write clearer descriptions, and fix obvious issues before a maintainer ever looks at it. The turning point for me was realizing the reviews weren’t just “nice to have.” They were protecting my reputation. If AI was writing a growing share of my code, I needed AI reviewing it with the same rigor, before it hit the repo. That’s, ultimately, why I joined CodeRabbit. The quality of the reviews, the codebase awareness, the ability to run checks locally in the IDE before pushing - it solved the exact bottleneck I was feeling in open source. AI code generation scaled output. AI code reviews scaled judgment. Open source didn’t need less AI. It needed AI in the right place in the workflow. And when reviews move left…before the pull request, before the human fatigue sets in - maintainers get their time back for what actually matters: design, direction, and shipping software that doesn’t wake you up at 2 a.m. ## Why CodeRabbit helps prevent OSS burnout Open source does not collapse because of bad code. It collapses because a handful of maintainers end up reviewing everything. AI made that imbalance worse. Contributors can now generate fixes, refactors, and feature PRs in minutes. Review still happens at human speed. The result is predictable: more notifications, longer queues, and maintainers spending nights and weekends doing unpaid triage. Burnout in open source rarely looks dramatic. It looks like this: * PRs sitting unreviewed for weeks. * Maintainers closing issues with “won’t fix” because they simply do not have time. * Fewer thoughtful reviews and more rubber stamps just to clear the backlog. * Talented maintainers quietly stepping back. That is not a tooling problem. It is an attention problem. CodeRabbit helps by acting as a first-pass reviewer that never gets tired and never rushes through a diff. It reviews every pull request line by line, checks for logic errors, edge cases, security risks, and cross-file impact, and flags issues before a human even opens the tab. For maintainers, that changes the shape of the work: * Instead of scanning for obvious bugs, you focus on architecture and intent. * Instead of manually checking for missed requirements, you validate higher-level decisions. * Instead of absorbing every small fix yourself, contributors address AI feedback before requesting review. ## CodeRabbit’s commitment to OSS CodeRabbit is committed to supporting open source software, having pledged [$1 million in sponsorships](https://www.coderabbit.ai/blog/coderabbit-commits-1-million-to-open-source). We rely heavily on open source technology and fundamentally believe that CodeRabbit's AI is a tool to assist, not replace, the vital role of open source maintainers. It absorbs the repetitive, error-prone parts of review so humans can spend their limited energy where it matters most. Open source survives on people. If AI is going to increase code volume, we need something equally consistent helping with review. Otherwise, the cost stays concentrated on the same few maintainers until they burn out. If you maintain an open-source project and your review queue keeps growing, [try CodeRabbit](https://www.coderabbit.ai/) on your next repository. CodeRabbit is free for open source. Every maintainer, contributor, and community can use our platform to cut through PR noise, automate code quality checks, and free up more time for meaningful contributions.

Developers are dead? Long live developers.

David Loker — Thu, 12 Feb 2026 00:00:00 GMT

Predictions about the end of programming are nothing new. Every few years, someone confidently announces that *this* time developers are **truly** finished. If you listened to these self-proclaimed Nostradamuses, devs were previously set to be replaced by everything from compilers ("If a machine writes the instructions, what's left for humans to do?") to low-code, no code tools ("Why hire developers when your VP can drag-and-drop an enterprise app?") to visual programming (“It looks like a flowchart, so it’s going to eliminate programming altogether). This time, the executioners are [AI coding agents](https://www.coderabbit.ai/guides/types-of-ai-coding-agents). Executives, founders, and tech influencers are lining up to tell the world that software engineers are living on borrowed time, that within a year or two, AI agents will write all the code, humans will step aside, and “developers” will join the long list of roles rendered obsolete. Like telephone switchboard operators or video rental clerks. They’re not entirely wrong. AI will indeed write much more of the code in a year or two, maybe even all of it for certain kinds of tasks. But rest assured, those sharing these takes are coming to the wrong conclusions. ## **“The king is dead. Long live the king.”** The phrase *“the king is dead, long live the king”* dates back to medieval Europe. It was announced at the moment of a monarch’s death to affirm continuity: one king has fallen, but the institution of kingship lives on in his successor. Applied to software engineering, the message is similar. The old model of developers is, indeed, dying out. In a few years, you won’t find devs who spend their days hand-writing syntax and carefully constructing every loop, import, and conditional. But the developer role itself isn’t disappearing. This isn’t an extinction event, it’s a succession. ## **It’s the end of developers as we know it (and you should feel fine)** The historical track record of “developers are finished” takes isn’t great. Every new abstraction triggers the same Twitter and media discourse: confident pronouncements, sweeping generalizations, and a level of certainty usually reserved for end-of-the-world cults. And yet, somehow, the world keeps running. In hindsight, these predictions look less prophetic and more like Silicon Valley’s version of Chicken Little. The story always ends the same way: declare developers dead, dramatically increase the demand for software, and then hire more developers than before. Only now, with better tools and bigger problems. Because the thing people forget is that each time abstraction rose, developers adapted. As **Grady Booch** put it on X recently: [![Developers are dead? Long live developers.](/content/assets/developers-are-dead-long-live-developers-inline-image.png)](https://x.com/Grady_Booch/status/2013331606795362398) Others have been more blunt. One widely shared take summed it up this way: [![Developers are dead? Long live developers.](/content/assets/developers-are-dead-long-live-developers-asset.png)](https://x.com/rough__sea/status/2013280952370573666) If you squint, these statements sound apocalyptic. But read carefully, and they’re actually saying something very different. They’re not claiming engineers stop existing. They’re claiming *how* software gets written is fundamentally changing. And they’re right. ## **What’s *actually* dying: Manual syntax production** AI is getting very good at generating code. Large chunks of application logic that once required careful human authorship can now be produced automatically. Boilerplate, glue code, scaffolding, even moderately complex algorithms are increasingly cheap to generate. That means some code simply won’t be written directly by humans anymore in the future as AI improves. But not all of it. Even in a world where AI coding bots could potentially take over your job completely, they likely wouldn’t be trusted with things that we prefer human judgement around like: * Critical systems * Performance-sensitive paths * Novel architectures * Ambiguous or underspecified domains These still demand deep human judgment. And even when AI writes the code, it doesn’t mean the human role disappears. It just shifts. ## **Shift 1: From writing code to *knowing what good looks like*** One tweet captured this change succinctly: [![Developers are dead? Long live developers.](/content/assets/developers-are-dead-long-live-developers-asset-2.png)](https://x.com/thiagotm/status/2013309376988131537?s=20) This is the uncomfortable truth about AI code generation: it is extremely confident, often persuasive, and occasionally very (elegantly) wrong. Catching those failures requires more, not less, expertise. You don’t review AI-generated code by just checking syntax. You review it by asking: * Does this match the intent? * Are the assumptions sound? * Does this fail safely? * What happens at the edges? * Is this maintainable six months from now? That’s not junior work, that’s senior judgment. And while we at CodeRabbit have created AI code reviews to help make the job of parsing AI code easier and catch bugs humans often miss, we don’t claim to be able to take a human out of the loop. A human developer is critical for catching business logic mistakes and other key nuances that AI isn’t able to. We just make their job of reviewing large volumes of AI generated code less overwhelming and, therefore, capable of scaling. ## **Shift 2: From coding, *then* reviewing to reviewing intent first** As AI systems take on more of the work of producing code, the most important human contribution shifts upstream. Code validation won’t start at the PR stage anymore, it will start before the code is actually written by reviewing the intent and plan. This is where prompt reviews are set to become central. With AI writing most of the code, developers will increasingly focus on things only they can do like: * Decomposing ambiguous problems * Understanding business goals or feature design * Specifying interfaces and acceptance criteria * Defining constraints and non-goals before anything is generated * Providing the right context at the right time * Building tight feedback loops * Designing for safety (security, privacy, reliability) * Setting success criteria that can be evaluated after the fact * Reviewing outcomes against original goals, not just surface correctness That’s a lot of work when teams have to keep pace with the speed of coding agents. And that involves being extremely clear about intent early on and ensuring alignment. After all, misunderstood intent or lack of alignment is what causes rework and delays at the PR review or testing stage. Ambiguity is what produces brittle systems. Poorly articulated goals are what lead AI to generate large volumes of code that look reasonable, behave correctly in the happy path… and quietly miss what actually matters. Prompt review is therefore set to become a key part of the SDLC in the next few years where alignment is validated *before* generation. That will involve: * Checking whether assumptions are explicit or merely implied * Making tradeoffs visible instead of accidental * Ensuring that “done” is defined, not guessed at * Validating that the output matches intent In that world, planning and review converge into a single responsibility: shaping the problem so the system has a chance of solving the right one. The developers who thrive won’t be the ones who can coax the most lines of code out of a model. They’ll be the ones who can express intent precisely, recognize when outputs drift from it, and intervene early as a team to get alignment, before small misunderstandings scale into large, expensive failures. ## Why we created CodeRabbit Issue Planner %[https://youtu.be/zHlgipben70?si=MUCX6v86faep6DmM] This shift toward an intent-first workflow isn’t theoretical for us. It’s what we’ve been seeing play out with teams using CodeRabbit. As AI accelerates code production, we’ve increasingly seen bottlenecks at the code review stage due to the volume of code AI was helping write and the number of issues it was adding to that code. Our recent study found that [AI added 1.7x more bugs](https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report) and issues to code than humans did. But, from what we saw, we believed that the real problem wasn’t just AI coding agents but how teams were working with those agents. Prompts were vague. Assumptions were implicit. Context lived in people’s heads or scattered Slack threads. Often AI was left to fill in the gaps and did so confidently but wrongly leading to more work at the review stage. What’s more, there was now a cold start problem where it became onerous and time consuming to draft the requirements into a prompt in a way that included all the assumptions, specs, and context so that AI could actually understand and properly execute on the code. That’s the problem that our CodeRabbit Issue Planner is designed to address. It supports developers in turning ideas into concrete, reviewable plans that are digestible by AI coding agents. It surfaces assumptions, clarifies scope, defines constraints, and makes tradeoffs explicit *before* generation begins. Importantly, Issue Planner doesn’t make judgment calls on behalf of developers like a coding agent might do if it made a plan itself. It doesn’t decide what “good” looks like. It doesn’t replace architectural thinking or product understanding. It creates an editable first draft of a prompt that can be reviewed by anyone on the team to create space for alignment and visibility earlier and to help support decisionmaking. That’s because as code generation becomes faster and cheaper, the value of developers concentrates around intent, judgment, and alignment. CodeRabbit’s planning product is built to support that reality. It helps developers focus more on judgement and strategy, helps catch misunderstandings sooner, and stays firmly in control of how they use AI in order to improve output and reduce rework later on in the development process. ## Developers aren’t going away. Somewhere right now, a CEO or two is likely polishing keynote slides about “the end of developers.” If history is any guide, they're going to be very disappointed. Not because AI won’t change software engineering (it already has) but because they’re mistaking a transformation for an extinction. What’s ending is a particular image of the developer: hunched over syntax, manually assembling every loop and conditional, measured by lines of code and hours spent typing. But the role of *developer* isn’t vanishing. It’s shedding a skin. The future developer: * Works at higher levels of abstraction * Defines intent rather than typing syntax * Reviews systems and outcomes, not just lines of code * Acts as the final arbiter of correctness, safety, and alignment As AI takes over the mechanics of production, more responsibility, not less, falls on the humans in the loop. Someone still has to decide what should be built, what constraints matter, what risks are acceptable, and whether the result actually solves the right problem. That work can’t be automated away, because it’s grounded in judgment, context, and accountability. So yes, developers, as we once knew them, are dying out. But long live developers. 👑 ***Interested in learning more about how we help devs in this new era?*** [***Try CodeRabbit reviews and Issue Planner today!***](https://coderabbit.link/8cjC8ui)

Issue Planner: collaborative planning for teams using coding agents

Parth Gandhi — Tue, 10 Feb 2026 00:00:00 GMT

For decades, the software development lifecycle has followed a familiar timeline. You create an issue, assign the work, manually write the code, get several peers to review it, test it, and ship. Each step took a relatively predictable amount of time and effort. You could find efficiencies but only to a degree. Then, AI coding agents came and exploded that timeline. At CodeRabbit, we’ve seen firsthand how AI shifted the backlog from writing code to reviewing and testing code. That’s why we launched our [PR Reviews product](https://www.coderabbit.ai/). When AI also created the need to review code locally while working on code with an agent, we launched our [IDE](https://www.coderabbit.ai/ide) and [CLI](https://www.coderabbit.ai/cli) code review tools. Lately, we’ve been seeing another problem. With coding agents in the loop, writing code is no longer the slowest or most critical part of the process. Planning is. Intent is. Alignment is. ## Why we’re taking on planning… (Hint: We see a lot of slop in reviews) The cost of being even slightly unclear about things like scope, assumptions, or success criteria has huge implications later on in the workflow when agents are involved… like in reviews where we were seeing a lot of slop and rework required 👀. But it also means needing to iterate on prompts until the agent gets it right and extensively reworking generated content. What’s more, creating effective prompts with the right context and intent is time consuming, creates issues with adoption of AI tools and generates uneven outcomes across teams due to differences in prompting skills and approach. Increasingly, developers are hungry for ways to streamline the planning process and to shift collaboration left by reviewing prompts before code is written. That’s why we’re **launching the beta for CodeRabbit Issue Planner.** We’re leveraging the same industry-leading context engine we use in our code reviews to help teams create plans, automate the drafting of prompts, and collaborate to ensure alignment. ## The challenge: Planning in the AI era ![Issue Planner: Collaborative planning for teams using coding agents](/content/assets/issue-planner-collaborative-planning-for-teams-with-ai-agents-inline-image.png) Today, teams spend more time prompting, re-prompting, and correcting AI-generated code than they ever expected. Prompts have grown longer as developers learned that being more specific helps increase output quality. Context is a big part of that but it has to be brought in from scattered docs, tickets, and Slack threads. If you don’t take the time to add it, agents make assumptions. Often not the right ones. The same goes with intent. Clear intent is the difference between your agent building the right thing instead of “something.” The result of these prompting challenges is rework, “AI slop,” and frustration. Good prompting is complicated, time consuming, and hard to do well. It requires: * Decomposing ambiguous problems * Specifying interfaces and acceptance criteria * Providing the right context at the right time * Building tight feedback loops * Designing for safety (including security, privacy, reliability) Meanwhile, on any team there’s always some teammates who know how to guide agents well and others who don’t. That creates uneven output quality, more review overhead, and stalled adoption. After all, if you have to rework the code every time because your prompt isn’t specific enough, it often seems like it’s faster to just write the code yourself. ## How our Issue Planner works %[https://youtu.be/zHlgipben70] Our Issue Planner is a new workflow that happens collaboratively within your issue tracker and helps teams define scope, assumptions, and success criteria together – before any code is written. While other tools have tried to tackle AI planning, they’ve done it in a disconnected way that doesn’t allow for team alignment or doesn’t bring in the right context. With AI coding agents, planning can no longer live in people’s heads, scattered tickets, or half-remembered conversations. It has to be explicit, shared, reviewable, and machine-readable. Instead of treating prompting as a solo activity, our Issue Planner turns planning into a collaborative, reviewable step in your SDLC. ### Issue Planner walkthrough 1) CodeRabbit Issue Planner integrates with your issue tracker to help you plan where issues live (currently available in **Linear, Jira, GitLab Issues, and GitHub Issues).** **2)** When an issue is created, CodeRabbit’s industry-leading context engine gets to work. It starts by creating a Coding Plan. That includes: * Sharing a high-level overview of how we will research and approach the task. * Pinpointing the files that need to change. * Breaking complex product requirements into multiple phases and tasks for an AI Coding Agent to create tasks. * Enriching each phase and task with context gathered from past issues, PRs, organizational context from tools like Notion, Confluence, and others. 3) This creates an editable, structured Coding Plan and consumable prompt package that can be executed on IDE or CLI tools or shared with a coding agent. You can give feedback, refine the assumptions the tool made, and regenerate the plan. 4) You and your team can review and refine the prompt before any code is written. Versions of the prompt are stored for future reference and use. 5) When ready, CodeRabbit hands off the prompt to the coding agent of your choice. This process saves you time, automates planning, improves your prompt output, and allows your team to align on intent in order to streamline the rest of the [software development lifecycle](https://www.coderabbit.ai/guides/agentic-sdlc). [Learn more in Docs](https://docs.coderabbit.ai/issues/planning) ## How our Issue Planner helps teams ![Issue Planner: Collaborative planning for teams using coding agents](/content/assets/issue-planner-collaborative-planning-for-teams-with-ai-agents-asset.png) For the last several months, Issue Planner has been available to select CodeRabbit customers. Teams using it saw immediate improvements. * **Accelerated workflows:** CodeRabbit highlights both what needs to change and how to change it, so teams spend less time orienting themselves and more time actually shipping. Instead of reverse-engineering intent from a ticket or PR, engineers and agents start with a clear plan. * **Better intent → better output:** Agents receive explicit requirements, assumptions, and constraints up front. That clarity translates directly into higher-quality output, code that matches team standards, respects architecture decisions, and solves the right problem the first time. * **Increased AI adoption:** By leveling the playing field around prompting and planning, CodeRabbit makes agents usable for the whole team, not just power users. Less reliance on “AI whisperers,” fewer bottlenecks, and more consistent results across engineers. * **Reduced rework:** Clear plans mean fewer back-and-forth prompt cycles, less cleanup, and fewer PRs that are *technically correct but functionally wrong*. Teams spend less time undoing AI output and more time moving forward. * **Less slop:** Better planning reduces hallucinations, spaghetti code, and invented requirements. When agents are grounded in real context and agreed-upon intent, output stays focused, readable, and maintainable. * **Real collaboration:** Planning doesn’t happen in isolation. CodeRabbit Issue Planner brings humans into alignment before agents execute. That way, decisions are shared, assumptions are visible, and the whole team agrees on the plan before any code is written. ## But… why Collaborative Planning and prompt reviews? ### Limitations of planning directly with agents Many AI coding tools encourage you to plan directly with an agent inside your editor, treating planning as a private, one-to-one interaction between a developer and an AI coding agent. In that scenario, requirements, assumptions, and constraints are only accessible to one person or are loosely implied in a prompt that no one else ever sees. AI coding agents also often don’t have the codebase context needed to create a comprehensive plan. When planning is rushed, under-specified or lacks proper context, agents do what they’re designed to do: fill in the gaps. The result is often code that looks coherent but quietly drifts from team standards, architectural decisions, or product intent. Without shared visibility or early feedback, misalignment isn’t discovered until a PR review, after time has already been spent generating, fixing, and reworking code. ### Why we chose Collaborative Planning That’s why we chose a different direction: planning that’s collaborative, reviewable, and shared, so teams can agree on intent before agents ever start writing code. Collaborative planning turns intent into a shared artifact the whole team can align on before execution begins. By reviewing and refining prompts together via prompt reviews, assumptions become explicit, constraints are clarified, and decisions are made deliberately rather than inferred. Agents stop guessing and start executing against clear instructions, producing output that’s more predictable, usable, and aligned with how the team actually builds software. In this model, prompting is no longer a personal skill for a few power-users. It becomes a team competency that scales, reduces rework, and allows agents to operate more reliably within agreed-upon boundaries. Another key benefit of planning collaboratively in your issue tracker also means that developers aren’t locked into one agent or editor. They can choose the coding agents that work best for them. ## What’s next? Roadmap for CodeRabbit Issue Planner We’re just getting started with our Issue Planner. We are building additional capabilities that will help developers collaborate more effectively, operationalize strong plans, and compound organizational knowledge over time. Here’s just two things to look forward to: * **Deeper collaboration and better prompt review workflows:** We’ll be expanding support for more team collaboration directly within Issue Planner, including the ability to add discussion threads, activity and decision logs, and explicit approval checkpoints for plans and prompts. The goal is to make intent review a shared, auditable process, closer to a Google Docs–style collaboration experience than a one-off prompt. * **Prompt repos that functions as “blueprints” for your code:** Our goal is to create a repo of the specs, design choices, and prompts that went into creating your codebase. This will function as a source of truth around the design directions and choices made and allow your team to more easily revisit decisions or tweak files without having to start from scratch. ## **Get started** ***CodeRabbit Issue Planner is available today in Linear, Jira, GitLab Issues, and GitHub Issues.*** [***Try it now.***](https://coderabbit.link/MW7YCkT)

Misalignment: The hidden cost of AI coding agents isn't from AI at all

Gur Singh — Tue, 10 Feb 2026 00:00:00 GMT

***TL;DR:*** *The real cost of AI agents isn’t tokens or tools; it’s misalignment that shows up as rework, slop, and slowed teams.* ## **The conversation everyone is having (and why it misses the point)** Most conversations about AI coding agents sound like a fantasy football draft. * Which model is better at autonomous coding? * Which one topped the benchmarks this week? * Which one “reasons” better? These debates take over blog posts, launch threads, and Slack channels everywhere. Devs on Twitter compare outputs line by line, argue over microscopic differences in reasoning quality, and swap models like they’re changing themes. Surely, *this* one will fix it. But for most teams, these differences aren’t what determine whether AI actually helps them ship faster. When AI-generated code goes sideways, it’s not typically because the model wasn’t smart enough. It’s because the agent had no idea what you really wanted. The code can be perfectly valid, logically sound, and honestly impressive… and still be completely wrong for your team, your codebase, or your product. That gap shows up as rework. Or repeated prompt tweaks. Or long review threads explaining intent after the fact. Or worse, developers spending more time correcting AI output than they would have spent writing the code themselves. Turns out, we’ve all been optimizing for model quality but forgot to measure drift. The most important factor in your success with AI agents isn’t which model you picked but how you’re adopting it as a team. Because turns out misalignment is the quiet, compounding problem that slowly eats your time while everyone’s busy arguing about benchmarks. ## **Speed exposed an existing problem** Before AI agents, misalignment was annoying… but survivable. Writing code took time. If requirements were fuzzy or assumptions were wrong, you usually discovered it halfway through writing the code or maybe during review. The feedback loop was slow, but forgiving. Humans hesitated, they asked questions, and then course-corrected as they went. But now? When an agent can generate hundreds or thousands of lines of code in seconds, it doesn’t stop to check whether your requirements were vague. It doesn’t ask clarifying questions. It doesn’t say, “Hey, this seems underspecified.” It just… goes. Confidently. In whatever direction it decides you likely intended. What used to be a small misunderstanding becomes a massive diff to review. What used to be a quick clarification turns into a full rewrite. And suddenly you’re staring at a PR thinking, “Technically, this is correct. Practically, it’s unusable.” This is why teams feel like AI made them both faster *and* slower. Execution is instant. Correction is not. The faster the agent, the more expensive unclear intent becomes. ## **The overlooked tax: AI rework** When AI output is bad, it rarely looks like failure. Instead, it looks like *iteration*. You run the agent. The result is close but not quite right. So, you tweak the prompt. Then, you tweak it again. You add more context. You clarify one edge case. You re-run the agent. And then you repeat. That’s progress these days. Each cycle feels small. But stack enough of them together and suddenly you’ve spent an hour rewriting prompts, reviewing generated code, and explaining intent without actually moving the work forward. It’s also a tax that can hit teams unevenly. Not everyone is an expert at writing prompts for coding agents. A few people on your team might be great at it but no teams are stacked entirely with prompting experts. Some might be prompting efficiently and effectively while others waste hours trying to rework things after the fact. Rework can show up as: * Long PR threads clarifying decisions that were never made explicitly * Code that technically passes tests but violates team conventions * Engineers quietly thinking, *“I could’ve just written this myself”* and deciding not to use AI the next time, reducing AI adoption across teams. The worse the alignment, the higher the tax. Every missing assumption gets paid for in re-prompts, reviews, and rewrites. Or worse, it leads to bugs and downtime in production. And unlike a failed build or a flaky test, this cost doesn’t always trip alarms. It just eats time quietly. ## **Misalignment doesn’t stay contained. It compounds.** Misalignment early in the workflow doesn’t just cause one problem: it causes a chain reaction. When intent isn’t clear up front, the agent fills in the gaps. That leads to code that’s *almost* right, close enough to look reasonable, but wrong enough to create work everywhere else. Suddenly, reviews are harder, tests are more complicated, and you’re having to rewrite half the PR yourself. What should’ve been a small clarification becomes a long review thread. What should’ve been a quick decision becomes a follow-up meeting. And what should’ve been a clean change becomes a series of patches trying to reconcile what the code does with what the team *meant*. Misalignment discovered early is cheap. Misalignment discovered late is expensive. Once code exists, every correction has a blast radius. You’re not just fixing intent, you’re undoing structure, refactoring assumptions, and explaining decisions that were never made explicitly by a human in the first place. AI doesn’t create this problem. It accelerates a problem that’s inherent in creating software: unclear intent and unaligned expectations. In the past, this was solved by a quick conversation to clarify. But because agents move so fast, they push misalignment downstream at full speed. By the time a human steps in, the cost has already multiplied. You’re no longer deciding *what* to build, you’re negotiating with code that already exists. ## **The real solution: Collaborative planning** ![Misalignment: The hidden cost of AI coding agents isn't from AI at all](/content/assets/issue-planner-collaborative-planning-for-teams-with-ai-agents-cover-image.jpg) Collaborative planning moves the hardest decisions to the moment when they’re still cheap: before code exists. You need to tackle them before agents start guessing and before assumptions get baked into hundreds or even thousands of lines of output. Instead of one person silently deciding what “good” looks like and encoding it into a prompt or worse, leaving it to the agent, teams agree on it upfront. Scope is explicit, assumptions are visible, and success criteria are shared. Intent stops living in someone’s head and starts living in an artifact the whole team can review. This changes the whole process. Agents stop improvising, reviews get lighter, and rework drops dramatically. Collaborative planning isn’t about slowing teams down or adding process. It’s about preventing the kind of misalignment that quietly drains time later. A few minutes of alignment upfront can save hours of cleanup downstream. ## **Why we built CodeRabbit Issue Planner** %[https://youtu.be/zHlgipben70] We didn’t wake up one day and decide to get into planning. We followed the failures. For years, CodeRabbit has lived where problems from [AI coding agents](https://www.coderabbit.ai/guides/types-of-ai-coding-agents) show up most clearly: in reviews. We’ve seen the same patterns repeat across teams, languages, and stacks, especially as AI agents became part of everyday workflows. The issues weren’t subtle. We saw generated code that technically worked but missed the point, long PR threads explaining decisions no one remembered making, and repeated fixes for assumptions that should’ve been surfaced earlier. In fact, our recent study found that they show up [1.7 times](https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report) more than in human written code. Again and again, the same conclusion kept surfacing: the problem didn’t start in the code. It started before the code existed. Reviewing output was no longer enough. If we wanted to reduce rework, improve [quality](https://www.coderabbit.ai/blog/code-reviews-made-easy-how-to-improve-code-quality), and actually help teams move faster with AI, we had to move upstream, without abandoning what made review effective in the first place. Collaborative planning gives teams a way to align on intent before agents start executing, so reviews catch fewer surprises and you can actually ship faster. ## How CodeRabbit Issue Planner works ![Misalignment: The hidden cost of AI coding agents isn't from AI at all](/content/assets/the-hidden-cost-of-ai-coding-agents-isnt-from-ai-at-all-asset.jpg) CodeRabbit Issue Planner connects directly to your issue tracker (we currently support Linear, Jira, and GitHub Issues) and helps teams [plan](https://www.coderabbit.ai/blog/issue-planner-collaborative-planning-for-teams-with-ai-agents) *before* any code is written. 1. When an issue is created, CodeRabbit’s [context engine](https://www.coderabbit.ai/blog/context-engineering-ai-code-reviews) automatically builds a **Coding Plan**. It outlines how the work should be approached, identifies which parts of the codebase are likely to change, and breaks complex requirements into clear phases and tasks that an AI coding agent can execute. 2. That plan is enriched with real [context](https://www.coderabbit.ai/guides/code-context): past issues and PRs, organizational knowledge from tools like Notion or Confluence, and relevant details pulled directly from your codebase. 3. The result is a structured, editable plan and a high-quality prompt package, ready to run in your IDE or via CLI tools. 4. Before anything is generated, your team can review the plan, refine assumptions, and iterate on the prompt together. Every version is saved, so decisions and intent don’t get lost over time. 5. When you’re ready, CodeRabbit hands the finalized prompt off to the coding agent of your choice. Then, generation starts with clarity, not guesswork. The outcome: faster planning, better prompts, fewer misunderstandings, and a development process aligned around shared intent from the very beginning. ## **Measure what actually matters** It’s easy to focus on more talked about parts of AI adoption. That’s why so many teams focus on things like: model quality, speed, and how impressive the output looks in isolation. But the teams that succeed with AI aren’t the ones chasing the agent with the best benchmark score. They’re the ones refining their processes to reduce rework, minimize confusion, and keep alignment cheap by planning effectively together. In the AI era, coding is becoming easier. The real work is shifting left towards planning and, at CodeRabbit, we’re here to help. ***Learn more and*** [***try Issue Planner today!***](https://coderabbit.link/kenhuQw)

We are committed to supporting open source: distributed $600,000 to open source maintainers in 2025

Santosh Yadav — Wed, 04 Feb 2026 00:00:00 GMT

CodeRabbit recognizes the growing need to support open source software (OSS), especially as AI accelerates the development landscape. While AI makes writing code faster and increases the frequency of pull requests, the time and effort of maintainers remain invaluable. Most open source projects rely on a small number of developers who manage these projects within their limited time. To address this, we provide OSS projects access to CodeRabbit for free. We’ve also made a commitment to giving the OSS community [$1 million in sponsorships.](https://www.coderabbit.ai/blog/coderabbit-commits-1-million-to-open-source) ## Open source funding is broken, we can fix it together At CodeRabbit, we view our support for open source as a vital investment in its future, rather than just "giving back." Open source software forms the foundation upon which most organizations build successful products. We rely on many OSS projects, including [**pnpm**](https://github.com/pnpm/pnpm), [**Biome**](https://github.com/biomejs/biome), and various React-based libraries. Our commitment is to ensure these projects remain sustainable for the long term, enabling widespread consumption and innovation. Critically, we want to make sure maintainers receive the necessary financial support to feel valued by the community and avoid burnout. Even with AI generating code, the use of OSS is not declining. In fact, as exemplified by the impact on major projects like [TailwindCSS](https://tailwindcss.com/) and [tldraw](https://www.tldraw.com/), AI is expected to significantly increase the download rates for widely used open source projects. We are proud to join the [open source pledge](https://opensourcepledge.com/join/) and formally commit to supporting open source software. ## CodeRabbit‘s effort in 2025 In 2025, CodeRabbit demonstrated its strong commitment to the open-source community by distributing over $600,000 in sponsorships, in addition to offering a free tier for open-source projects. This significant funding is part of our effort to support maintainers who continue to spend valuable time reviewing pull requests, a burden that is often exacerbated by the ongoing influx of AI-generated contributions to OSS projects. Further details on how CodeRabbit distributed more than $600,000 to various projects are available below. ![We are committed to supporting open source: Distributed $600,000 to open](/content/assets/we-are-committed-to-supporting-open-source-distributed-600000-to-open-source-maintainers-in-2025-inline-image-2.png) Part of our commitment to open source is sponsoring developers on GitHub, where we were able to support 18% of the maintainers for the packages CodeRabbit relies on directly. We're planning to boost that support even more in 2026. In the first quarter of 2026 we are giving away $100,000 to the tools we and members of our community rely on. ![We are committed to supporting open source: Distributed $600,000 to open](/content/assets/we-are-committed-to-supporting-open-source-distributed-600000-to-open-source-maintainers-in-2025-inline-image.png) We sponsored 74 devs and distributed around $267,000 using [GitHub sponsors](https://github.com/orgs/coderabbitai/sponsoring) and [open collective](https://opencollective.com/coderabbit). We directly collaborated with maintainers to distribute the remaining $360,000+ in funding. Our commitment to open source continues as we launch the [CodeRabbit oss program](https://docs.google.com/forms/d/e/1FAIpQLScBYzbvjENJLHnMreturAwXZI_90mUPIBonseala1ZAcTeOGw/viewform), which is part of our commitment to giving [$1 million to the open source community](https://www.coderabbit.ai/blog/coderabbit-commits-1-million-to-open-source).

An (actually useful) framework for evaluating AI code review tools

David Loker — Fri, 09 Jan 2026 00:00:00 GMT

Benchmarks promise clarity. They’re supposed to reduce a complex system to a score, compare competitors side by side, and let the numbers speak for themselves. But, in practice, they rarely do. Benchmarks don’t measure “quality” in the abstract. They measure whatever the benchmark designer chose to emphasize, under the specific constraints, assumptions, and incentives of the evaluation. Change the dataset, the scoring rubric, the prompts, or the evaluation harness, and the results can shift dramatically. That doesn’t make benchmarks useless, but it does make them fragile, manipulable, and easy to misinterpret. Case in point: database benchmarks. ### Database benchmarks: A *cautionary* tale The history of database performance benchmarks is a useful example. As benchmarks became standardized, vendors learned how to optimize specifically for the test rather than for real workloads. Query plans were hand-tuned, caching behavior was engineered to exploit assumptions, and systems were configured in ways no production team would realistically deploy. Over time, many engineers stopped trusting benchmark results, treating them as marketing signals rather than reliable indicators of system behavior. ### AI code review benchmarks are on the same trajectory We’re currently seeing AI code review benchmarks go down a similar path. As models are evaluated on curated PR sets, synthetic issues, or narrowly defined correctness criteria, tools increasingly optimize for benchmark performance rather than for the messy, contextual, high‑stakes reality of real code review. The deeper problem is not just that benchmarks can be misleading, it’s that many “ideal” evaluation designs are difficult to execute correctly in real engineering environments. When an evaluation framework is too detached from real workflows, too easy to game by badly configuring your competitor’s tool, or too complex to run well, the results become hard to trust. What follows below is a practical framework for ***effectively*** evaluating AI code review tools that balances rigor with feasibility, and produces results that are both meaningful and interpretable. ## **Start from your objectives and make them explicit** ![An (actually useful) framework for evaluating AI code review tools](/content/assets/framework-for-evaluating-ai-code-review-tools-inline-image.png) Before assembling datasets or choosing metrics, it’s critical to define what you actually care about. “Better code review” means different things to different teams, and an evaluation that doesn’t encode those differences will inevitably optimize for the wrong outcome. Common objectives include: * Catching real defects and risks before merge * Improving long‑term maintainability and reducing technical debt * Avoiding low‑value noise that degrades review quality * Maintaining developer trust and adoption It’s also important to distinguish between **leading indicators** and **lagging indicators**. Outcomes like fewer production incidents or higher long‑term throughput are real and important, but they often emerge over months, not weeks. Shorter evaluations should focus on signals that correlate strongly with those outcomes, such as the quality of issues caught, whether they are acted on, and how developers respond to the tool. Explicitly ranking your objectives such as quality impact, precision, developer experience, and throughput, helps ensure that your evaluation answers the questions that actually matter to your organization. ## Determine what kind of evaluation is needed ![An (actually useful) framework for evaluating AI code review tools](/content/assets/framework-for-evaluating-ai-code-review-tools-asset.png) The most reliable evaluation of any tool involves a real-world pilot over a controlled offline benchmark. This allows you to see how it works in day-to-day situations versus just evaluating a tool based on criteria defined by a third party vendor. ### In-the-wild pilot The most reliable signals come from observing how a tool behaves in real, day‑to‑day development. Real‑time evaluation reflects actual constraints: deadlines, partial context, competing priorities, and human judgment. It shows not just what a tool can detect in theory, but what it surfaces in practice, and whether those issues matter enough for developers to act on them. For this, select a few teams or projects for each tool and run each tool for a period of time under normal usage. Measure things like: * Real-world detection of issues. * Severity of issues caught. * Developer satisfaction and perceived utility. If possible, design A/B style experiments so you can measure using the tool vs no tool on comparable teams or repos or Tool A vs Tool B on similar workloads, perhaps alternating weeks or branches. ### Offline benchmark For teams that want additional confidence, controlled detection comparisons can provide useful insight if you design it yourself using your own pull requests and criteria so it gives you the data you actually need. However, it’s not required in most cases since it doesn’t provide as much useful data as a pilot and can be time intensive to set up. One practical approach is to use a private evaluation or mirror repository. A small, representative set of pull requests can be replayed, allowing multiple tools to be run on the same diffs without disrupting real workflows. These comparisons are best used to understand coverage differences by severity and category, and to identify systematic strengths and blind spots across tools. After that, you just need to compute the metrics you’re looking to track. For example: * Precision/recall by severity and issue type. * Comment volume and distribution. ## **Why evaluating multiple tools on the same pull request is usually misleading** If you want to do a head-to-head comparison via either a benchmark or a pilot, a common instinct is to run them all on the same exact pull requests rather than mirroring that PR and running each tool you’re comparing separately on it or running them on different but comparable PRs. On the surface, running them all simultaneously feels fair and efficient. In practice, it introduces serious problems. **When multiple AI reviewers comment on the same PR:** Human reviewers are overwhelmed with feedback and cognitive load spikes. No single tool can be experienced as it was designed to be used in that case. For example, some tools skip comments if they see another tool has already made that comment leading to the perception that that tool hasn’t found the issue. Review behavior changes—comments are skimmed, bulk‑dismissed, or ignored This creates interference effects. Tools influence each other’s perceived usefulness, and attention, not correctness, becomes the limiting factor. Precision metrics degrade because even high‑quality comments may be ignored simply due to volume. That makes it harder to know the percentage of comments your team would accept from each individual tool under normal usage. The result is that you lose the ability to evaluate usability, trust, workflow fit, and real‑world usefulness. You are no longer measuring how a tool performs in practice, but how reviewers cope with noise. Running multiple tools on the same exact PR can be useful in narrow, controlled contexts, such as offline detection comparisons, but it is a poor way to evaluate the actual experience and value of a code review tool. To understand whether a tool helps your team, it often best be experienced in isolation within a normal review workflow. ## Structuring fair comparisons without complex infrastructure There are practical ways to compare tools without building elaborate experimentation harnesses. **Parallel evaluation across repos or teams** is often the simplest approach. Select repos or teams that are broadly comparable in language, domain, and PR volume, and run different tools in parallel. Keep configuration effort symmetric and analyze results using normalization techniques (discussed below). Alternatively, **time‑sliced evaluation within the same repo or team** can work when parallel groups are not available. Run one tool for a defined period, then switch. This approach requires acknowledging temporal effects—release cycles, workload changes, learning effects—but can still produce useful, directional insights when interpreted carefully. Finally, simply **mirroring PRs and running reviews on them with separate tools** also works well, if you want to compare comments on the same PRs. In all these cases, the goal is to preserve a clean developer experience while collecting comparable data. In practice, these approaches can also be combined if a team feels like that’s helpful to give them a better idea of how a tool works. Teams may start with parallel evaluation across different repositories or teams, then swap tools after a fixed period. This helps balance differences in codebase complexity or workload over time, while still avoiding the disruption and interference that comes from running multiple tools on the same pull request. As with any time-based comparison, results should be normalized and interpreted with awareness of temporal effects, but this hybrid approach often provides a good balance of fairness, practicality, and interpretability. ## **Metrics that produce interpretable results** ![An (actually useful) framework for evaluating AI code review tools](/content/assets/framework-for-evaluating-ai-code-review-tools-asset-2.png) Based on successful deployments across thousands of repositories, we've identified a framework of seven metric categories that provide a complete picture of your integration which we suggest as metrics to measure to our customers. Each category answers a specific question about your AI implementation: 1. **Architectural Metrics** – Is the tool appropriately integrated? How many of an org’s repos are connected, how many extensions are they using (git, IDE, CLI). 2. **Adoption Metrics** – Are developers actually using it? These metrics include monthly active users (MAU), the percentage of total repositories covered and week-over-week growth. 3. **Engagement Metrics** – Are they just ignoring it or actively collaborating with it? These metrics include PRs reviewed versus Chat Sessions initiated. Also track “Learnings used,” how often the AI applies context from previous reviews to new ones. 4. **Impact Metrics** – Is it catching bugs that matter to the team? These metrics include number of issues detected, actionable suggestions, and the “acceptance rate” (percentage of AI comments that result in a code change). 5. **Quality & Security Metrics** – Is it preventing expensive bugs and security vulnerabilities? These metrics include Linter/SAST findings, security vulnerabilities caught (e.g., Gitleaks), and reduction in pipeline failures. 6. **Governance Metrics** – Is it enforcing standards across the team? These metrics include usage of pre-merge checks, warnings vs. errors, and implementation of custom governance rules. 7. **Developer Sentiment** – Are the developers happy with their experience and product? These metrics include survey results, qualitative feedback, and “aha” moments. ### **Accepted issues as a primary quality signal** Not all metrics are equally informative and some are far easier to misread than others. A practical evaluation should focus more attention on signals that are both meaningful and feasible to measure. One of the strongest indicators of value is whether a tool’s feedback leads to real action. An issue can reasonably be considered **accepted** when: * A subsequent commit addresses the comment or thread * A reviewer explicitly acknowledges that the issue has been resolved This behavioral signal captures correctness, relevance, and usefulness in a way that pure scoring metrics cannot. Accepted issues should be reported by: * **Severity** (e.g., critical, major, minor, low, nitpick) * **Category** (security, logic, performance, maintainability, testing, etc.) Both absolute counts and rates are informative, especially when interpreted together. ### **Precision and signal‑to‑noise** Acceptance rate (accepted issues relative to total surfaced) is a practical proxy for precision. On its own, it is insufficient; paired with comment volume, it becomes far more meaningful. High comment volume with low acceptance is a clear signal of noise. Patterns of systematically ignored categories or directories often reveal where configuration or tuning is needed. It’s also important to avoid the “LGTM trap.” That means a tool that leaves very few comments, all correct, may appear precise while missing large classes of issues. In many cases, broad coverage combined with configurability is preferable to narrow precision that cannot be expanded. ### **Coverage and issue discovery in real review flows** In typical workflows, the sequence is: PR opens → AI review → issues fixed → human review Because humans review after the tool, it is often impossible to say with certainty which issues humans would have caught independently. Instead of trying to infer counterfactuals precisely, focus on practical signals: * Accepted issues that led to substantive code changes * Accepted issues in categories humans historically miss (subtle logic, edge cases, maintainability) * Consistent patterns of issues surfaced across PRs Sampling can help here. Reviewing a subset of PRs and asking, “Would this issue likely have been caught without the tool?” is often more informative than attempting exhaustive labeling. ### **Normalization: Making comparisons fair** Raw counts are misleading when pull requests vary widely in size and complexity. Normalization is essential for fair comparison. Useful normalization dimensions include: * PR size (lines changed, files touched) * PR type (bug fix, feature, refactor, infra/config, test‑only) * Domain or risk area (frontend/backend, high‑risk components) Comparisons should be made within similar buckets, and distributions are often more informative than averages. Small samples at extremes should be interpreted cautiously. ## **Interpreting throughput and velocity** Throughput metrics like time‑to‑merge are easy to misread. When a tool begins catching real issues that were previously missed, merge times may initially increase. This often reflects improved rigor rather than reduced productivity. Throughput should therefore be treated as a secondary metric, normalized by PR complexity and evaluated over time alongside quality indicators. Short‑term slowdowns can be a leading indicator of long‑term gains in code health. ## **Bringing it all together** ![An (actually useful) framework for evaluating AI code review tools](/content/assets/framework-for-evaluating-ai-code-review-tools-asset-3.png) A reliable evaluation does not require perfect benchmarks or elaborate experimental design. It requires clarity about objectives, careful interpretation of metrics, and an emphasis on real‑world behavior. Start with normal workflows and behavioral signals. Normalize to make comparisons fair. Use controlled comparisons selectively to deepen understanding. Combine quantitative metrics with concrete examples of impact ## **Final takeaway** Benchmarks are useful starting points, not verdicts. The most trustworthy evaluations of AI code review tools are grounded in real workflows, user behavior‑based signals, and balance rigor with practicality. When done well, they provide confidence not just that a tool performs well on paper, but that it meaningfully improves both the immediate quality of code changes and the long‑term health of the codebase. ***Curious how CodeRabbit performs on your codebase? Get a*** [***free trial today!***](https://coderabbit.link/QjWcnUj)

Why users shouldn’t choose their own LLM models: choice is not always good

David Loker — Fri, 09 Jan 2026 00:00:00 GMT

Giving users a dropdown of LLMs to choose from often seems like the right product choice. After all, users might have a favorite model or they might want to try the latest release the moment it drops. One problem: unless they’re an ML engineer running regular evals and benchmarks to understand where each model actually performs best, that choice is liable to hurt far more than it helps. You end up giving users what they think they want, while quietly degrading the quality of what they produce with your tool with inconsistent results, wasted tokens, and erratic model behavior. For example, developers may unknowingly pick a model that’s slower, less reliable for their specific task, or tuned for a completely different kind of reasoning pattern. Or they might choose a faster model than they need that won’t comprehensively reason through the task. Choosing which model to use isn’t a matter of personal taste… It's a systems-level optimization problem. The right model for any task depends on measurable performance across dozens of task dimensions, not just how recently it was released or how smart users perceive it to be. And that decision should belong to engineers armed with eval data, not end users who wrongly believe they’ll get better results with the model they personally prefer. ## **The myth of *‘preference’* in AI model selection** Many AI platforms love to market model choice as a premium feature. “Choose GPT-4o, Claude, or Gemini” sounds empowering and gives users the impression that they will get the best or latest experience. It taps into the same instinct that makes people want to buy the newest phone the week it launches: the feeling that *newer* and *bigger* must mean *better*. The reality, though, is that most users have no idea which model actually performs best for their specific use case. And even if they did, that answer would likely shift from one query to another. The “best” model for code generation might not be the “best” for bug detection, documentation, or static analysis. There might also be multiple models that are best at different parts of a code review or other task, depending on what kind of code is being reviewed. Some tasks require greater creativity and reasoning depth; others need precision and consistency. A developer who blindly defaults to “the biggest model available” for coding help, often ends up with slower, more expensive, and less deterministic results. In some cases, a smaller, domain-tuned model will handily outperform its heavyweight cousin. ## **Why model selection is an *evaluation* problem, not preference** ![Why users shouldn’t choose their own LLM models: Choice is not always good](/content/assets/why-users-shouldnt-choose-their-own-llm-models-choice-is-not-always-good-inline-image.png) Model selection isn’t a matter of taste… it's a data problem. Behind the scenes, engineers run thousands of evaluations across tasks like code correctness, latency, context retention, and tool integration. These aren’t one-time benchmarks; they’re continuous systems designed to measure how models actually perform under specific, reproducible conditions. The results form a kind of performance map which shows which model excels at refactoring versus summarizing code or which one handles long-context reasoning without drifting off-topic. End users never see that map. While some might read benchmarks or articles about a model’s performance, most are making decisions blind, guided mostly by hunches, Reddit posts, or vague impressions of “smartness.” Even if they wanted to, users rarely have the time or infrastructure to run their own evals across hundreds of tasks and models. The result is that people often optimize for hype rather than outcomes… choosing the model that feels cleverest or sounds more fluent, not the one that’s objectively better for the job. And human perception alone is a terrible way to evaluate model competence. A model that seems chatty and confident can be consistently wrong, while one that feels hesitant might actually deliver the most accurate, reproducible results. Without hard data from evaluations, those distinctions disappear. ## **The prompting paradox** ![Why users shouldn’t choose their own LLM models: Choice is not always good](/content/assets/why-users-shouldnt-choose-their-own-llm-models-choice-is-not-always-good-asset.png) One critical drawback to choosing your own model is that no two LLMs *think* alike. Each model interprets prompts slightly differently. Some are more literal, others more associative; some favor verbosity, others prefer minimalism. A prompt that works perfectly on GPT-5 might completely derail on Sonnet 4.5, leading to hallucinated code, missing context, or an output that ignores key constraints. Temperature, context length, and formatting differences only make the problem worse. A model with a higher temperature parameter might produce creative explanations but rewrite variable names, while another with stricter formatting rules could break markdown or indentation. These small mismatches can quietly poison a workflow, especially in environments where consistent structure matters most like with code reviews, diff comments, or documentation summaries. When users choose their own models, they unknowingly disrupt the prompt-engineering assumptions that keep those workflows stable in systems where the prompts are written for the user. Every prompt is tuned with certain expectations about how the model parses instructions, handles errors, and formats its output. Swap out the model and those assumptions collapse. It’s even harder to navigate in situations where the user writes the prompt themselves, like with AI coding tools. Users rarely have enough context, knowledge, and experience to write effective prompts for each model. However, over time, they might find a few prompting methods that help them get the best out of a particular model. If they later change to a new model, they often find their old prompts aren’t as effective and need to learn from scratch trying to get the best results from that new model. That’s why well-designed systems rely on **model orchestration**, not user preference. In review pipelines or agentic systems, predictability is everything. You need each component to behave consistently so downstream tools and other models can interpret the results. Giving users the freedom to swap models isn’t customization; it’s chaos engineering without the safety net. ## **The hidden costs of model freedom** Once users can switch models at will, all the invisible consistency that makes AI-assisted workflows dependable begins to crumble. The consequences aren’t abstract; they’re measurable and they multiply fast. Across teams, the first thing you notice is inconsistency. Two developers can run the same review prompt and get completely different feedback. One gets a precise diff comment, the other might get a philosophical musing on the meaning of clean code. That inconsistency makes it impossible to reproduce results, which is deadly for any process that relies on traceability or QA. Then there’s cost. Larger models burn through tokens faster and often respond slower, introducing both financial waste and latency drag. And when users unknowingly pick models with shorter context windows, the result is truncated inputs or missing context. It’s like asking someone to summarize a novel after reading only half of it. ## **The better alternative: Dynamic, data-driven routing** The smarter alternative to user-driven chaos is dynamic, data-driven routing. That means systems that automatically choose the right model for the right task. Instead of asking users to guess which LLM might perform best, auto-routing engines make that choice in real-time based on metrics, evals, and historical performance. Think of it as orchestration, not selection. A large model might be routed in for creative reasoning, open-ended problem solving, or complex code explanations. A smaller, domain-tuned model might handle deterministic checks, linting, or static analysis where precision and speed matter more than eloquence. The system continuously evaluates the outcomes tracking correctness, latency, and user feedback in order to refine its routing logic over time. This approach turns what used to be human guesswork into an adaptive, evidence-based process. The routing system learns which models excel at which tasks, under which conditions, and how to balance cost, speed, and quality. Advanced teams already operate this way. In **CodeRabbit**, for example, the orchestration layer sits between the user and the models, using structured prompts, eval data, and performance histories to dispatch requests intelligently. Developers don’t have to think about which LLM is behind a particular review comment. The system has already chosen the optimal one, validated against internal benchmarks. In short, dynamic routing makes model choice invisible. The user gets consistently high-quality results; the engineers get measurable control and efficiency. Everyone wins. Except the dropdown menu. ## **Expertise is in the system, not the slider** The takeaway here is simple: model selection isn’t a feature, it’s a quality control issue. The best results come from systems that make those choices invisibly and are grounded in data, not gut instinct. When model routing is automatic and performance-based, users get consistent, high-quality outputs without needing to think about which model is doing the work. Every product that puts a “Choose your LLM” dropdown front and center is outsourcing an engineering decision to the least equipped person to make it. Or, put another way: **the best AI tool UI is no LLM dropdown at all.** ***Curious what it looks like when an AI pipeline optimizes for LLM fit?*** [***Try CodeRabbit for free today!***](https://coderabbit.link/br1LHRx)

CodeRabbit's AI code reviews now support NVIDIA Nemotron

Sahil Mohan Bansal — Mon, 05 Jan 2026 00:00:00 GMT

TL;DR: Blend of frontier & open models is more cost efficient and reviews faster. NVIDIA Nemotron is supported for CodeRabbit self-hosted customers. We are delighted to share that CodeRabbit now supports the [NVIDIA Nemotron](https://developer.nvidia.com/nemotron?ncid=pa-srch-goog-405472&_bt=785763502016&_bk=nvidia%20nemotron&_bm=p&_bn=g&_bg=194843200048&gad_source=1&gad_campaignid=23296574832&gbraid=0AAAAAD4XAoH92Ikfem4gRmk2l43ihyoMZ&gclid=CjwKCAiAjc7KBhBvEiwAE2BDOUJ675w2dEUGzWfCsu26BnfRGvO40xVPRh1PPGasRkAvJUjOmPDHYxoCDf4QAvD_BwE) family of open models among its blend of Large Language Models (LLMs) used for AI code reviews. Support for Nemotron 3 Nano has initially been enabled for CodeRabbit’s self-hosted customers running its [container image](https://docs.coderabbit.ai/self-hosted/github) on their infrastructure. Nemotron is used to power the context gathering and summarization stage of the code review workflow before the frontier models from OpenAI and Anthropic are used for deep reasoning and generating review comments for bug fixes. ## How Nemotron helps: Context gathering at scale This new blend of open and frontier models allows us to improve the overall speed of context gathering and improves cost efficiency by routing different parts of the review workflow to the appropriate model family, while delivering review accuracy that is at par with running frontier models alone. High quality AI code reviews that can find deep lying and hidden bugs require lots of context gathering related to the code being analyzed. The most frequent (and most token-hungry) work is summarizing and refreshing that context: what changed in the code and does it match developer intent, how do those changes connect with rest of the codebase, what are the repo conventions or custom rules, what external data sources are available to aid the review, etc. This context building stage is the workhorse of the overall AI code review process and it is run several times iteratively throughout the review workflow. [NVIDIA Nemotron 3 Nano](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16) was built for high-efficiency tasks and its large context window (1 million tokens) along with fast speed helps to gather a lot of data and run several iterations of context summarization and retrieval. ![CodeRabbit architecture with Nemotron support](/content/assets/coderabbit-ai-code-reviews-now-support-nvidia-nemotron-inline-image.png) ## **Blend of frontier and Open Models** When you open a Pull Request (PR), CodeRabbit’s code review workflow is triggered starting with an isolated and secure sandbox environment where CodeRabbit analyzes code from a clone of the repo. In parallel, CodeRabbit pulls in context signals from several sources: * Code and PR index * Linter / Static App Security Tests (SAST) * Code graph * Coding agent rules files * Custom review rules and Learnings * Issue tickets (Jira, Linear, GitHub issues) * Public MCP servers * Web search To dive deeper into our context engineering approach you can check out our blog: [The art and science of context engineering for AI code reviews](https://www.coderabbit.ai/blog/the-art-and-science-of-context-engineering). A lot of this context, along with the code diff being analyzed, is used to generate a PR Summary before any review comments are generated. This is where open models come in. Instead of sending all of the context to frontier models, CodeRabbit now uses Nemotron Nano v3 to gather and summarize the relevant context. Summarization is at the heart of every code review and is the key to delivering high signal-to-noise in the review comments. After the summarization stage is completed the frontier models (e.g., OpenAI GPT-5.2-Codex and Anthropic Claude-Opus/Sonnet 4.5) perform deep reasoning to generate review comments for bug fixes, and execute agentic steps like review verification, pre-merge checks, and “finishing touches” (including docstrings and unit test suggestions). ## What this means for our customers CodeRabbit is now enabling Nemotron-3-Nano-30B support (initially for its self-hosted customers) for the context summarization part of the review workflow along with the frontier models from OpenAI and Anthropic. This results in faster code reviews without compromising quality. We are also delighted to support the [announcement from NVIDIA](https://blogs.nvidia.com/blog/open-models-data-tools-accelerate-ai) today about the expansion of its Nemotron family of open models and are excited to work with the company to help accelerate AI coding adoption across every industry. [Get in touch](https://www.coderabbit.ai/contact-us/sales) with our team to access CodeRabbit’s container image if you would like to run AI code reviews on your self-hosted infrastructure.

Our new report: AI code creates 1.7x more problems

David Loker — Wed, 17 Dec 2025 00:00:00 GMT

*What we learned from analyzing hundreds of open-source pull requests.* Over the past year, AI coding assistants have gone from emerging tools to everyday fixtures in the development workflow. At many organizations, a part of every code change is now machine-generated or machine-assisted. But while this has been accelerating the speed of development, questions have been quietly circulating: * Why are more defects slipping through into staging? * Why do certain logic or configuration issues keep appearing? * And are these patterns tied to AI-generated code? It would appear like AI is playing a significant role. [A recent report](https://go.cortex.io/rs/563-WJM-722/images/2026-Benchmark-Report.pdf?version=0) found that while pull requests per author increased by 20% year-over-year, thanks to help from AI, incidents per pull request increased by 23.5%. This year also brought several high-visibility incidents, postmortems, and anecdotal stories pointing to AI-written changes as a contributing factor. These weren’t fringe cases or misuses. They involved otherwise normal pull requests that simply embedded subtle mistakes. And yet, despite rapid adoption of AI coding tools, there has been surprisingly little concrete data about how AI-authored PRs differ in quality from human-written ones. So, CodeRabbit set out to answer that question empirically in our [**State of AI vs Human Code Generation Report.**](https://www.coderabbit.ai/whitepapers/state-of-AI-vs-human-code-generation-report) ## Our State of AI vs Human Code Generation Report We analyzed **470 open-source GitHub pull requests**, including **320 AI-co-authored PRs** and **150 human-only PRs**, using CodeRabbit’s structured issue taxonomy. Every finding was normalized to issues per 100 PRs and we used statistical rate ratios to compare how often different types of problems appeared in each group. The results? Clear, measurable, and consistent with what many developers have been feeling intuitively: **AI accelerates output, but it also amplifies certain categories of mistakes.** [READ THE FULL REPORT](https://www.coderabbit.ai/whitepapers/state-of-AI-vs-human-code-generation-report) ### Limitations of our study Getting data on issues that are more prevalent in AI-authored PRs is critical for engineering teams but the challenge was determining which PRs were AI-authored vs human authored. Since it was impossible to directly confirm authorship of each PR of a large enough OSS dataset, we checked for signals that a PR was co-authored by AI and assumed that those that didn’t have it were human authored, for the purposes of the study. This resulted in statistically significant differences in issue patterns between the two datasets, which we are sharing in this study so teams can better know what to look for. However, we cannot guarantee all the PRs we labelled as human authored were actually authored only by humans. Our full methodology is shared at the end of the report. ## **Top 10 findings from the report** No issue category was uniquely AI but most categories saw significantly more errors in AI-authored PRs. That means, humans and AI make the same kinds of mistakes. AI just makes many of them more often and at a larger scale. ### **1\. AI-generated PRs contained ~1.7× more issues overall.** Across 470 PRs, AI-authored changes produced **10.83 issues per PR**, compared to **6.45** for human-only PRs. Even more striking: high-issue outliers were much more common in AI PRs, creating heavy review workloads. ### **2\. Severity escalates with AI: More critical and major issues.** ![Our new report: AI code creates 1.7x more problems](/content/assets/state-of-ai-vs-human-code-generation-report-asset.png) AI PRs show ~**1.4–1.7×** more critical and major findings. ### **3\. Logic and correctness issues were 75% more common in AI PRs.** These include business logic mistakes, incorrect dependencies, flawed control flow, and misconfigurations. Logic errors are among the most expensive to fix and most likely to cause downstream incidents. ### **4\. Readability issues spiked more than 3× in AI contributions.** ![Our new report: AI code creates 1.7x more problems](/content/assets/state-of-ai-vs-human-code-generation-report-asset-2.png) The single biggest difference across the entire dataset was in readability. AI-produced code often looks consistent but violates local patterns around naming, clarity, and structure. ### **5\. Error handling and exception-path gaps were nearly 2× more common.** ![Our new report: AI code creates 1.7x more problems](/content/assets/state-of-ai-vs-human-code-generation-report-asset-3.png) AI-generated code often omits null checks, early returns, guardrails, and comprehensive exception logic, issues tightly tied to real-world outages. ### **6\. Security issues were up to 2.74× higher** ![Our new report: AI code creates 1.7x more problems](/content/assets/state-of-ai-vs-human-code-generation-report-asset-4.png) The most prominent pattern involved improper password handling and insecure object references. While no vulnerability type was unique to AI, nearly all were amplified. ### **7\. Performance regressions, though small in number, skewed heavily toward AI.** ![Our new report: AI code creates 1.7x more problems](/content/assets/state-of-ai-vs-human-code-generation-report-asset-5.png) Excessive I/O operations were ~8× more common in AI-authored PRs. This reflects AI’s tendency to favor clarity and simple patterns over resource efficiency. ### **8\. Concurrency and dependency correctness saw ~2× increases.** Incorrect ordering, faulty dependency flow, or misuse of concurrency primitives appeared far more frequently in AI PRs. These were small mistakes with big implication ### **9\. Formatting problems were 2.66× more common in AI PRs.** Even teams with formatters and linters saw elevated noise: spacing, indentation, structural inconsistencies, and style drift were all more prevalent in AI-generated code. ### **10\. AI introduced nearly 2× more naming inconsistencies.** Unclear naming, mismatched terminology, and generic identifiers appeared frequently in AI-generated changes, increasing cognitive load for reviewers. [READ THE FULL REPORT](https://www.coderabbit.ai/whitepapers/state-of-AI-vs-human-code-generation-report) ## Why these patterns appear Why are teams seeing so many issues with AI-generated code? Here’s our analysis: * **AI lacks local business logic:** Models infer code patterns statistically, not semantically. Without strict constraints, they miss the rules of the system that senior engineers internalize. * **AI generates surface-level correctness:** It produces code that looks right but may skip control-flow protections or misuse dependency ordering. * **AI doesn’t adhere perfectly to repo idioms:** Naming patterns, architectural norms, and formatting conventions often drift toward generic defaults. * **Security patterns degrade without explicit prompts:** Unless guarded, models recreate legacy patterns or outdated practices found in older training data. * **AI favors clarity over efficiency:** Models often default to simple loops, repeated I/O, or unoptimized data structures. ## What engineering teams can do about it Adopting AI coding tools isn’t simply about speeding up development. It requires rethinking the guardrails that ensure all code entering production is safe, maintainable, and correct. Based on the patterns in the data, here are the most important takeaways for teams: ### **1\. Give AI the context it needs** AI makes more mistakes when it lacks business rules, configuration patterns, or architectural constraints. Provide prompt snippets, repo-specific instruction capsules, and configuration schemas to reduce misconfigurations and logic drift. ### **2\. Use policy-as-code to enforce style** Readability and formatting were some of the biggest gaps. CI-enforced formatters, linters, and style guides eliminate entire categories of AI-driven issues before review. ### **3\. Add correctness safety rails** Given the rise in logic and error-handling issues: * Require tests for non-trivial control flow * Mandate nullability/type assertions * Standardize exception-handling rules * Explicitly prompt for guardrails where needed ### **4\. Strengthen security defaults** Mitigate elevated vulnerability rates by centralizing credential handling, blocking ad-hoc password usage, and running SAST and security linters automatically. ### **5\. Nudge the model toward efficient patterns** Offer guidelines for batching I/O, choosing appropriate data structures, and using performance hints in prompts. ### **6\. Adopt AI-aware PR checklists** Reviewers should explicitly ask: * Are error paths covered? * Are concurrency primitives correct? * Are configuration values validated? * Are passwords handled via the approved helper? These questions target the areas where AI is most error-prone. ### 7\. Get help reviewing and testing AI code Code review pipelines weren’t created to handle the higher volume of code teams are currently shipping with the help of AI. Reviewer fatigue has been [found to lead to more issues and missed bugs](https://smartbear.com/resources/case-studies/cisco-systems-collaborator/). An AI code review tool like CodeRabbit helps by standardizing code reviews acts as a third-party source of truth that standardizes quality across different AI tools that teams might use while reducing the time and cognitive labor needed for reviews. That allows developers to concentrate on reviewing the more complex parts of the code changes and reduce the amount of bugs and issues that end up in production. [READ THE FULL REPORT](https://www.coderabbit.ai/whitepapers/state-of-AI-vs-human-code-generation-report) ## The bottom line AI coding tools are powerful accelerators, but acceleration without guardrails increases risk. [Our analysis](https://www.coderabbit.ai/whitepapers/state-of-AI-vs-human-code-generation-report) shows that AI-generated code is **consistently more variable**, **more error-prone**, and **more likely to introduce high-severity issues** without the right protections in place. The future of AI-assisted development isn’t about replacing developers. It’s about building systems, workflows, and safety layers that amplify what AI does well while compensating for what it tends to miss. For the teams that want the speed of AI without the surprises, the data is clear: **Quality isn’t automatic. It requires deliberate engineering. Even when using AI tools.** ***An AI code review tool could also help.*** [***Try CodeRabbit today.***](https://app.coderabbit.ai/login?free-trial=)

Behind the curtain: What it really takes to bring a new model online at CodeRabbit

David Loker — Fri, 05 Dec 2025 00:00:00 GMT

***When we published*** [***our earlier article***](https://www.coderabbit.ai/blog/the-end-of-one-sized-fits-all-prompts-why-llm-models-are-no-longer-interchangeable?) ***on why users shouldn't choose their own models, we argued that model selection isn't a matter of preference, it's a systems problem. This post explains exactly why.*** Bringing a new model online at CodeRabbit isn't a matter of flipping a switch; it's a multi-phase, high-effort operation that demands precision, experimentation, and constant vigilance. Every few months, a new large-language model drops with headlines promising “next-level reasoning,” “longer context,” or “faster throughput.” For most developers, the temptation is simple: plug it in, flip the switch, and ride the wave of progress. We know that impulse. But for us, adopting a new model isn’t an act of curiosity, it’s a multi-week engineering campaign. Our customers don’t see that campaign, and ideally, they never should. The reason CodeRabbit feels seamless is precisely because we do the hard work behind the scenes evaluating, tuning, and validating every model before it touches a single production review. This is what it really looks like. ### **1\. The curiosity phase: Understanding the model’s DNA** ![Behind the curtain: What it really takes to bring a new model online at](/content/assets/behind-the-curtain-what-it-really-takes-to-bring-a-new-model-online-at-coderabbit-inline-image.png) Every new model starts with a hypothesis. We begin by digging into what it claims to do differently: is it a reasoning model, a coding model, or something in between? What’s its architectural bias, its supposed improvements, and how might those capabilities map to our existing review system? We compare those traits against the many model types that power different layers of our context-engineering and review pipeline. The question we ask isn’t, “is this new model better?” but, *“where might it fit?”* Sometimes it’s a candidate for high-reasoning diff analysis; other times, for summarization or explanation work. Each of those domains has its own expectations for quality, consistency, and tone. From there, we start generating experiments. Not one or two, but dozens of evaluation configurations across parameters like temperature, context packing, and instruction phrasing. Each experiment feeds into our evaluation harness, which measures both quantitative and qualitative dimensions of review quality. ### **2\. The evaluation phase: Data over impressions** ![Behind the curtain: What it really takes to bring a new model online at](/content/assets/behind-the-curtain-what-it-really-takes-to-bring-a-new-model-online-at-coderabbit-inline-image-2.png) This phase takes time. We run models across our internal evaluation set, collecting hard metrics that span coverage, precision, signal-to-noise, and latency. These are the same metrics that underpin the benchmarks we’ve discussed in earlier posts like [*Benchmarking GPT-5*](https://www.coderabbit.ai/blog/benchmarking-gpt-5-why-its-a-generational-leap-in-reasoning), [*Claude Sonnet 4.5: Better Performance, but a Paradox*](https://www.coderabbit.ai/blog/claude-sonnet-45-better-performance-but-a-paradox), [*GPT-5.1: Higher signal at lower volume*](https://www.coderabbit.ai/blog/gpt-51-for-code-related-tasks-higher-signal-at-lower-volume?), and [*Opus 4.5: Performs like the systems architect.*](https://www.coderabbit.ai/blog/opus-45-for-code-related-tasks-performs-like-the-systems-architect?) But numbers only tell part of the story. We also review the generated comments themselves by looking at reasoning traces, accuracy, and stylistic consistency against our current best-in-class reviewers. We use multiple LLM-judge recipes to analyze tone, clarity, and helpfulness, giving us an extra lens on subtle shifts that raw metrics can’t capture. If you’ve read our earlier blogs, you already know why this is necessary: models aren’t interchangeable. A prompt that performs beautifully on GPT-5 may completely derail on Sonnet 4.5. Each has its own “prompt physics.” Our job is to learn it quickly and then shape it to behave predictably inside our system. ## **3\. The adaptation phase: Taming the differences** ![Behind the curtain: What it really takes to bring a new model online at](/content/assets/behind-the-curtain-what-it-really-takes-to-bring-a-new-model-online-at-coderabbit-asset.png) Once we understand where a model shines and where it struggles, we begin tuning. Sometimes that means straightforward prompt adjustments such as fixing formatting drift or recalibrating verbosity. Other times, the work is more nuanced: identifying how the model’s internal voice has changed and nudging it back toward the concise, pragmatic tone our users expect. We don’t do this by guesswork. We’ll often use LLMs themselves to critique their own outputs. For example: “This comment came out too apologetic. Given the original prompt and reasoning trace, what would you change to achieve a more direct result?” This meta-loop helps us generate candidate prompt tweaks far faster than trial and error alone. During this period, we’re also in constant contact with model providers, sharing detailed feedback about edge-case behavior, bugs, or inconsistencies we uncover. Sometimes those conversations lead to model-level adjustments; other times they inform how we adapt our prompts around a model’s quirks. ## **4\. The rollout phase: From lab to live traffic** ![Behind the curtain: What it really takes to bring a new model online at](/content/assets/behind-the-curtain-what-it-really-takes-to-bring-a-new-model-online-at-coderabbit-asset-2.png) When a model starts to perform reliably in offline tests, we move into phased rollout. First, we test internally. Our own teams see the comments in live environments and provide qualitative feedback. Then, we open an early-access phase with a small cohort of external users. Finally, we expand gradually using a randomized gating mechanism so that traffic is distributed evenly across organization types, repo sizes, and PR complexity. Throughout this process, we monitor everything: * Comment quality and acceptance rates * Latency, error rates, and timeouts * Changes in developer sentiment or negative reactions to CodeRabbit comments * Precision shifts in suggestion acceptance If we see degradation in any of these signals, we roll back immediately or limit exposure while we triage. Sometimes it’s a small prompt-level regression; other times, it’s a subtle style drift that affects readability. Either way, we treat rollout as a living experiment, not a switch-flip. ## **5\. The steady-state phase: Continuous vigilance** Once a model is stable, the work doesn’t stop. We monitor it constantly through automated alerts and daily evaluation runs that detect regressions long before users do. We also listen, both to our own experience (we use CodeRabbit internally) and to customer feedback. That feedback loop keeps us grounded. If users report confusion, verbosity, or tonal mismatch, we investigate immediately. Every day, we manually review random comment samples from public repots that use us to ensure that quality hasn’t quietly slipped as the model evolves or traffic scales. ## **6\. Why we do all this & why you shouldn’t have to** ![Behind the curtain: What it really takes to bring a new model online at](/content/assets/behind-the-curtain-what-it-really-takes-to-bring-a-new-model-online-at-coderabbit-asset-3.png) Each new model we test forces us to rediscover what “good” means under new constraints. Every one comes with its own learning curve, its own failure modes, its own surprises. That’s the reality behind the promise of progress. Could an engineering team replicate this process themselves? Technically, yes. But it would mean building a full evaluation harness, collecting diverse PR datasets, writing and maintaining LLM-judge systems, defining a style rubric, tuning prompts, managing rollouts, and maintaining continuous regression checks. All of this before your first production review! That’s weeks of work just to reach baseline reliability. And you’d need to do it again every time a new model launches. We do this work so you don’t have to. Our goal isn’t to let you pick a model; it’s to make sure you never have to think about it. When you use CodeRabbit, you’re already getting the best available model for each task, tuned, tested, and proven under production conditions. Because “choosing your own model” sounds empowering until you realize it means inheriting all this complexity yourself. ## **Takeaway** Model adoption at CodeRabbit isn’t glamorous. It’s slow, meticulous, and deeply technical. But it’s also what makes our reviews consistent, trustworthy, and quietly invisible. Every diff you open, every comment you read, is backed by this machinery. Weeks of evaluation, thousands of metrics, and countless prompt refinements all in service of one thing: Delivering the best possible review, every time, without you needing to think about which model is behind it. Try out CodeRabbit today. [Get a free 14-day trial!](https://app.coderabbit.ai/login?free-trial=)

It's harder to read code than to write it (especially when AI writes it)

Aleks Volochnev — Thu, 04 Dec 2025 00:00:00 GMT

*"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it."* * Brian Kernighan (co-creator of Unix and co-author of *The C Programming Language*) I've been programming since I was ten. When it became a career, I got obsessed with code quality: clean code, design patterns, all that good stuff. My pull requests were polished like nobody's business: well-thought-out logic, proper error handling, comments, tests, documentation. Everything that makes reviewers nod approvingly. Then, LLMs came along and changed everything. I don't write that much code anymore since AI does it faster. Developer’s work now mainly consists of two parts: explaining to a model what you need, then verifying what it wrote, right? I’ve become more of a code architect and quality inspector rolled into one. And here came a problem I knew all too well from my years as a tech lead: ## **READING CODE IS ACTUALLY HARDER THAN WRITING IT.** As an open-source maintainer and senior developer, I had to review tons of other people's code, and I learned what Kernighan said the hard way. Reading unfamiliar code is exhausting. You have to reverse-engineer someone else's thought process, figure out why they made certain decisions, and consider edge cases they might have missed. With my own code, reviewing and adjusting were a no-brainer. I designed it, I wrote it, and the whole mental model was still fresh in my head. Now the code is coming from an LLM and suddenly reviewing "my own code" has become reviewing someone else's code. Except this "someone else" writes faster than I can think and doesn't take lunch breaks. AI is supposed to help, but if I want to ship production-grade software now, I actually have more hard work to do than before. The irony! And that’s why, for my first blog post since joining CodeRabbit, I wanted to focus on that fact. This is also, incidentally, why I decided to join CodeRabbit. But we’ll get to that part later. ## **We’re human (unfortunately for code quality)** Here's where things get uncomfortable: we're human beings, not code-reviewing machines. And human brains don't want to do hard work, thoroughly reviewing something that a) already runs fine, b) passes all the tests, and c) someone else will review anyway. It's so much easier to just git commit && git push and go grab that well-deserved coffee. Job is done! I went from “writing manually and shipping quality code,” to “generating code fast but shipping… bad code!” The quality dropped not because I had less time as I actually had MORE time since I wasn't typing everything myself. I just tend to “shorten” this verification phase, telling myself "it works, the tests pass, the team will catch anything major." ## **The problem with "Catching it in review"** At this point, I was already using CodeRabbit to review my team's pull requests (as an OSS-focused dev, I was an early adopter), and those reviews were genuinely helpful! CodeRabbit would catch things that slipped through. Security issues, edge cases, some logic bugs. Those problems that are easy to miss when you're moving fast. But here's the thing: those reviews were coming too late. The code was already pushed. Already in the repository, visible to the entire team. Sure, CodeRabbit would flag the issues and I'd fix them but not before my teammates had seen my AI-generated code with obvious problems that I didn't bother to review properly. That's not a great look when you've spent decades building a reputation for quality. ## **Enter: CodeRabbit in an IDE** Then, I discovered CodeRabbit had an IDE extension. The AI code reviewer I was already using for PRs could also review my code locally, before anything hits the repo. This was exactly what I needed. When I ask CodeRabbit to check or simply stage my changes, CodeRabbit reviews them right in VS Code, catching issues before git push. Now, my team sees only the polished version, just like the old days. Except now, I'm shipping AI-generated code at AI speeds. And I’m doing it with actual quality control. Automatic reviews mean no willpower required: I don't have to remember to run it, I don't have to open a separate tool. It just happens at commit time. Reviewing doesn't feel like plowing in the rain anymore. ![It's harder to read code than to write it (especially when AI writes it)](/content/assets/its-harder-to-read-code-than-to-write-it-especially-when-ai-writes-it-asset.png) This gets critical when you're looking at potential security headaches, like the one on the screenshot. CodeRabbit caught an access token leak that could've been a total disaster! Issues like this needs to be addressed *before* that code gets pushed to a repository. ![It's harder to read code than to write it (especially when AI writes it)](/content/assets/its-harder-to-read-code-than-to-write-it-especially-when-ai-writes-it-asset-2.png) More than that, when it finds something, the fixes are committable. The tool doesn’t tell me to "go figure it out" but gives actual suggestions I can apply immediately, in one click. ![It's harder to read code than to write it (especially when AI writes it)](/content/assets/its-harder-to-read-code-than-to-write-it-especially-when-ai-writes-it-asset-3.png) For more advanced cases that can’t be resolved with a simple fix, CodeRabbit IDE extension writes a prompt that it sends to an AI agent of your choice. Fun fact: CodeRabbit is so good in writing prompts so I got a lot to learn from, improving my Prompt Engineering skills! ![It's harder to read code than to write it (especially when AI writes it)](/content/assets/its-harder-to-read-code-than-to-write-it-especially-when-ai-writes-it-asset-4.png) Even the free CodeRabbit IDE Review plan offers incredibly helpful feedback and catches numerous issues. However, the Pro plan unlocks its true power, providing the same comprehensive coverage you expect from regular CodeRabbit Pull Request reviews: tool runs, Code Graph analysis, and much more - there is a huge infrastructure behind every check! ![It's harder to read code than to write it (especially when AI writes it)](/content/assets/its-harder-to-read-code-than-to-write-it-especially-when-ai-writes-it-asset-5.png) ## **The bottom line** Brian Kernighan was right: reading code is harder than writing it. That was true in 1974 and it's even more true now when AI can generate 300 lines while you're still thinking about a variable name. We thought AI would make our jobs easier. And it does… if you only count the writing. But the reading verifying, reviewing, and understanding what the AI agent actually built? That got harder. Many of us are doing 10x the volume at 10x the speed, which means 10x more code to read with the same human brain that gets lazy and wants coffee breaks. The solution isn't to slow down or go back to typing everything manually. The solution is to automate the code review process as thoroughly as we automated the code writing process. If your AI writes the code, another AI should be reading it before you get to it. The quality of the reviews is why I recently transitioned from being a CodeRabbit user to joining the team. And that’s why you should also try [CodeRabbit in your IDE](https://docs.coderabbit.ai/code-editors). The free tier means there's basically no excuse not to try it. Your reputation will thank you. ***Get started today with a*** [***14-day free trial!***](https://app.coderabbit.ai/login?free-trial=)

Gemini 3 for code-related tasks: The dense engineer

David Loker — Wed, 26 Nov 2025 00:00:00 GMT

**TL;DR:** It doesn’t just write patches; it writes a complete argument for every change. When Gemini 3 is right, it’s spectacularly right. When it’s wrong, it still sounds right. ### **Every model writes in our house style. Gemini 3 rewrites the rules.** ![Gemini 3 for code-related tasks: The dense engineer](/content/assets/gemini-3-for-code-related-tasks-the-dense-engineer-inline-image-4.png) All of CodeRabbit’s models follow the same structural blueprint: a short headline, an explanation, and a patch. However, Gemini 3 uses that frame differently. It fills every inch of space with evidence, preconditions, and causal reasoning. Each review reads like a technical brief wrapped around a diff. %[https://youtu.be/ou1sNnSeM4E?si=WRfez1XIkIJ8F955] Gemini 3 is confident, detailed, and relentlessly specific. Its comments read like they were written by a senior engineer who wants to fix the issue and demonstrate *why* the fix is necessary. That density is its defining trait. Every comment feels significant, even the ones you might not ultimately act on. ## **Benchmarking context for Gemini 3** ![Here is how CodeRabbit benchmarked Gemini 3](/content/assets/gemini-3-for-code-related-tasks-the-dense-engineer-inline-image-5.png) We evaluated Gemini 3 using CodeRabbit’s standard benchmark: 25 pull requests seeded with known error patterns (EPs) across C++, Java, Python, and TypeScript. Each comment was scored by multiple LLM judges and hand-validated by our engineers. We measured precision, important-share, and signal-to-noise ratio (SNR), the same metrics used in all of our model evaluations. We also assessed tone, length, and style, since how a model *communicates* can affect whether developers accept its suggestions. ![Gemini 3 vs GPT-5.1 and Opus 4.5 and Sonnet 4.5](/content/assets/gemini-3-for-code-related-tasks-the-dense-engineer-inline-image.png) **Interpretation:** Gemini 3 sits in the middle of the group for precision but provides excellent real-bug coverage. Roughly three of every four comments are important (critical or major). Its SNR of 3.2 puts it close to Opus 4.5 in reliability, but Gemini 3 expresses itself with greater conviction and detail. ### **Style and Tone of Gemini 3** ![Style and tone of Gemini 3 vs GPT-5.1 and Opus 4.5 and Sonnet 4.5](/content/assets/gemini-3-for-code-related-tasks-the-dense-engineer-inline-image-3.png) **Tone summary:** Gemini 3 is the most assertive of the four. It communicates with confidence, and for the most part, that confidence is justified. Even when it makes a mistake, the comment sounds credible enough to make developers stop and re-check the code, which may add to its practical value but may be confusing for some. ![Gemini 3 Style and tone](/content/assets/gemini-3-for-code-related-tasks-the-dense-engineer-inline-image-2.png) ### **The dense engineer personality of Gemini 3** Gemini 3 compresses an exceptional amount of reasoning into compact comments. The average comment is only 16 lines long, yet each one unpacks a complete causal chain: *what broke, why it happened, and how to fix it.* For example, in a concurrency issue found in a C++ worker pool, Gemini 3 doesn’t simply say “missing lock.” It reconstructs the sequence: unlock, wait, missed signal, dead thread. Then it provides a single-line patch that resolves the race. In another TypeScript review, the model identifies that MAX\_SAFE\_INTEGER disables cache eviction, explains the performance risk, and proposes an LRU fallback. These are not stylistic suggestions. These are corrections that improve program reliability. This combination of density and accuracy defines Gemini 3’s personality. Every comment is an argument, and most of those arguments hold up under scrutiny. ### **What Gemini 3 feels like in review** ![Gemini 3 reviews](/content/assets/gemini-3-for-code-related-tasks-the-dense-engineer-asset.png) Reading Gemini 3 feels distinct from reading any other reviewer. It is confident and structured, but its reasoning is particularly dense. Each comment reads like a detailed technical review from a lead engineer who insists on context before approving a change. Comments often open with a directive such as “Fix this race condition,” then expand into a clear explanation, referencing specific files and ending with a patch. The result feels like an expert walking you through both the problem and the fix. Developers describe Gemini 3 as a reviewer that sounds sure of itself and provides evidence to support its claims. Its direct tone can feel intense, especially compared to GPT-5.1’s measured precision or Opus 4.5’s calm logic. However, each comment feels like a mini design review, explaining what to change, why it matters, and what trade-offs exist. The model’s high information density requires careful reading, but it provides proportionate insight. Even when wrong, Gemini 3 often reveals something valuable about hidden edge cases or architectural assumptions. ### **What the numbers show** **1\. Density correlates with correctness.** Longer comments (top half by length) pass more often, with 53% precision compared to 34% for shorter comments. Important comments (critical or major) average **847 characters**, nearly twice the size of unimportant ones (**442**). When Gemini 3 takes time to elaborate, it is typically accurate. **2\. Tone tracks severity.** Assertiveness rises with severity: **92%** of major and **67%** of critical comments are assertive, while only **36%** of minor comments are. Hedging increases to **36%** on minor issues. The model uses its strongest voice for the most serious problems, which makes it effective for triage. **3\. Confidence correlates with quality.** Assertive comments pass more frequently (**47.6%**) than neutral (**36%**) or hedged (**33%**) ones. Gemini 3’s confidence is generally supported by evidence rather than overstated. **4\. Patches indicate reliability.** When a diff or code block appears, the precision rate improves. Over **70%** of assertive comments contain diffs, compared with only **17%** of hedged comments. The presence of a patch often signals that the model’s reasoning is grounded in the actual code. ### **What it gets right** ![Gemini 3 for code-related tasks: The dense engineer](/content/assets/gemini-3-for-code-related-tasks-the-dense-engineer-asset-2.png) Gemini 3’s strongest areas are concurrency and system correctness. It frequently detects interleaving and synchronization issues that other models overlook. It excels at diagnosing the *why* behind a bug: * **Thread-safety:** Describes lost wakeups and inconsistent locking with narrative precision, then offers a concise patch. * **Lifecycle management:** Identifies missing shutdown hooks or unclosed resources and recommends explicit cleanup. * **Algorithmic stability:** Corrects comparator logic and off-by-one ranges to restore invariants. * **System configuration:** Finds default values that disable expected behavior and recommends practical limits. Each of these shows how Gemini 3’s detailed reasoning leads to direct, verifiable fixes. ### **When it overreaches** Gemini 3’s conviction can occasionally go too far. On stylistic or low-severity issues, it may overstate importance. Some comments labeled “Critical” are actually minor or aesthetic. Its assertive tone can make small findings sound urgent. This model performs best when paired with experienced reviewers who can distinguish critical bugs from overconfident advice. Even its overreaches tend to highlight genuine inefficiencies or readability concerns. Few comments are without value. ### **Why density matters** Gemini 3 trades brevity for understanding. It does not simply provide a fix; it delivers a short investigation. This depth makes it particularly valuable for large, complex systems. Precision measures whether a model hits the target, but density measures how much a developer learns by reading it. In production environments, that difference matters. A concise reviewer like GPT-5.1 delivers quick, targeted notes. Gemini 3, by contrast, provides comprehensive reasoning that increases confidence and reduces the likelihood of missing subtle defects. Let's put it this way, *“You don’t skim Gemini 3. You study it.”* ### **Practical guidance: When to use Gemini 3** | **Use it when...** | **Because...** | | --- | --- | | Concurrency-heavy or resource-sensitive code | It excels at identifying synchronization and lifecycle issues. | | Depth over brevity | Longer, more detailed comments correlate with accuracy. | | You need actionable patches | Around 65% of comments include ready-to-apply diffs. | | You can manage assertive tone | Its confidence is helpful but occasionally overstated. | | You are mentoring newer developers | Each comment serves as both a fix and an educational note. | Gemini 3 is not ideal for superficial or stylistic reviews. It is best used when precision, explanation, and insight are more important than speed. ### **Closing thoughts: The shape of Gemini 3’s reasoning** Gemini 3 does more than fix code; it presents a logical case for every fix. Each comment is a complete story of cause, effect, and resolution. When it is correct, it feels like reading a senior engineer’s deep-dive analysis. Even when it is wrong, it provides insight into how to think about the problem. **Takeaway**: If GPT-5.1 is the decisive teammate and Opus 4.5 the disciplined architect, Gemini 3 is the dense engineer who delivers a fully reasoned diff that is confident, comprehensive, and intent on proving its point. ***Want to try out CodeRabbit?*** [***Get at 14-day free trial!***](https://coderabbit.link/PZ2w3mZ)

Opus 4.5 for code-related tasks: performs like the systems architect

David Loker — Mon, 24 Nov 2025 00:00:00 GMT

## **Every model reasons. Opus 4.5 audits.** Every new model arrives with the same promise: smarter reasoning, cleaner code, and better answers. But Opus 4.5 from Anthropic doesn’t just reason; it *audits*. It reads code as if returning to a system it helped design, identifying weak points and refining architecture. Where other models narrate their logic or prescribe surgical fixes, Opus 4.5 performs structured, systematic reviews that feel more like technical documentation than conversation. We integrated Opus 4.5 into CodeRabbit’s benchmark harness to understand what makes this model distinct. The result was not higher raw intelligence or flashier prose, but discipline. This model doesn’t just find bugs; it *builds context* around them. It treats review as an engineering process, rather than a guessing game. ![Opus 4.5 vs. GPT 5.1 and Sonnet 4.5](/content/assets/opus-45-for-code-related-tasks-performs-like-the-systems-architect-inline-image.png) ## **Benchmarking context for Opus 4.5** ![Opus 4.5 benchmarking context](/content/assets/opus-45-for-code-related-tasks-performs-like-the-systems-architect-inline-image-4.png) At CodeRabbit, we evaluate new LLMs using a controlled benchmark of **25 complex pull requests** seeded with known error patterns (EPs) across C++, Java, Python, and TypeScript. Each comment generated by a model is scored by an LLM judge for three key factors: * **Precision:** Whether it correctly identifies the EP. * **Important-share:** The percentage of comments that are genuinely critical or major (real bugs, not style issues). * **Signal-to-noise ratio (SNR):** The ratio of important to unimportant comments. Our evaluation framework, refined over multiple generations of models, combines automated LLM judging with **hand validation** to ensure accuracy. We also use **multiple judges and repeated trials** to measure consistency and understand variance. Each iteration improves the process through better prompts, refined labeling, and expanded coverage, resulting in more reliable outcomes. ## **Scoreboard (Actionable comments only)** ![Opus 4.5 vs GPT 5.1 and Sonnet 4.5](/content/assets/opus-45-for-code-related-tasks-performs-like-the-systems-architect-inline-image-3.png) **What this means:** Opus 4.5 sits between Sonnet 4.5’s high-volume, verbose style and GPT-5.1’s lean, surgical precision. It delivers higher per-comment precision and a greater share of meaningful findings than Sonnet 4.5. While it recorded one fewer EP pass (15 vs. 16), that difference falls within normal variance. In several runs, Opus 4.5 matched or even surpassed both GPT-5.1 and Sonnet 4.5. The takeaway is a model that balances signal, structure, and coverage with consistent reliability. ## **Style and tone of Opus 4.5** ![Opus 4.5 vs GPT 5.1 and Sonnet 4.5](/content/assets/opus-45-for-code-related-tasks-performs-like-the-systems-architect-inline-image-2.png) Opus 4.5’s reviews are structured, concise, and focused. With assertiveness around 33% and hedging near 15%, its tone reads as measured and professional. The balance of tone and density gives it an analytical voice that feels practical and confident. The high use of code blocks and diff patches underscores its bias toward action; it talks less and *edits more*. ![Opus 4.5 style and tone ](/content/assets/opus-45-for-code-related-tasks-performs-like-the-systems-architect-asset.png) ## **Structured intelligence: Predictable form and cross-language consistency** Opus 4.5’s comments follow an architectural rhythm of headline, rationale, and diff. Nearly 80% include code blocks, and most conclude with a concise patch. Each resembles a clear bug report that specifies cause, effect, and resolution. This structure holds across languages. Whether reviewing C++, Java, Python, or TypeScript, the cadence remains consistent, averaging 19 lines and 790 characters per comment. This uniformity simplifies automation and enhances readability. It also makes Opus 4.5 feel like a single engineer’s consistent voice across an entire codebase. * *C++ (WorkerThreadPool):* Detects a lost wakeup race with a three-step interleaving and a one-line diff fix. * *Java (OrderService):* Flags a missing volatile on a double-checked lock and provides the corrected pattern. * *Python (Batch client):* Replaces a synchronous HTTP client with an asynchronous equivalent to prevent blocking calls. * *TypeScript (Cache manager):* Identifies that Number.MAX\_SAFE\_INTEGER disables eviction and suggests realistic defaults. These are concise, code-native insights, each actionable and grounded in sound reasoning. ## **The confidence inversion** Opus 4.5’s tone is balanced but occasionally reveals a subtle inversion: when it is wrong, it can sound slightly more certain. Although the model is generally measured, this behavioral quirk means tone alone is not always a reliable indicator of correctness. To account for this, we pair tone data with correctness metrics in evaluation summaries to maintain consistent calibration. Opus 4.5 rarely speculates; it simply explains, even when it’s wrong. ## **System-level reasoning: Fixing context, not just code** While most models target the immediate defect, Opus 4.5 focuses on the surrounding system. Its recommendations frequently adjust lifecycles, add safety checks, or refine defaults. Examples: * **TypeScript Cache:** Rewrites eviction logic, adds TTL enforcement, and updates defaults to prevent silent OOM. * **Java OrderService:** Replaces HashMap with ConcurrentHashMap and identifies missing ExecutorService shutdown. * **Python Client Lifecycle:** Adds explicit shutdown hooks for long-lived async clients. * **C++ FileAccessEncrypted:** Resolves a validation bug that blocked all encrypted files and improves upstream error handling. These are not single-line fixes but systemic corrections. The model treats code as an interconnected ecosystem rather than a collection of isolated issues. ## **Cost, effort & efficiency** Anthropic’s Effort parameter provides direct control over how deeply the model reasons. In high-effort mode, Opus 4.5 explores every dependency path. In medium-effort mode, it trims reasoning depth to save tokens. Even with high-effort reasoning, its reviews averaged about 25% fewer output tokens than Sonnet 4.5, balancing higher per-token costs ($25 per million output tokens) with greater efficiency. This disciplined structure pays for itself by producing fewer digressions and maintaining consistent clarity. ## **What it feels like to read Opus 4.5** ![Reviews of Opus 4.5](/content/assets/opus-45-for-code-related-tasks-performs-like-the-systems-architect-asset-2.png) If Sonnet 4.5 feels like a teacher and GPT-5.1 like a decisive teammate, Opus 4.5 is the **architect reviewing your PR**. Its tone is calm and deliberate, never commanding. It assumes you understand the domain and aims to confirm the details. The result is feedback that reads like peer review from a systems engineer: consistent, structured, and quietly authoritative. ### **Tone and personality** Opus 4.5’s voice is measured and analytical. It rarely uses dramatic language or unnecessary severity. Instead, it conveys certainty through order, concise summaries, specific evidence, and focused corrections. The tone builds trust, delivering feedback that feels like it comes from a mentor familiar with your system. ### **Depth vs. density** Its comments are compact yet informative. When an issue warrants detailed explanation, Opus 4.5 delivers it without excess. For simpler problems, it resolves them with brief, precise advice. This balance of detail and brevity keeps reviews readable and comprehensive. ### **Flow and readability** The model’s structural rhythm of context, cause, and correction allows developers to scan quickly while retaining meaning. Developers often describe its comments as “structured snapshots” that tell a short, self-contained story: what happened, why it matters, and how to fix it. ### **Practical impact and developer trust** Because Opus 4.5 avoids inflated confidence and theatrical phrasing, developers trust it more readily. It comes across as confident yet professional, firm but not forceful. When it errs, it sounds like a reasoned hypothesis instead of an overreach. That restraint, more than precision alone, makes its reviews feel *professionally human*. Each comment reads like a design note. It states the invariant that failed, proposes a patch, and explains the rationale inline. The clarity is high enough that many of its comments could be pasted directly into changelogs or issue trackers without revision. ## **Strengths and weaknesses of Opus 4.5** ![Opus 4.5 strengts and weaknesses](/content/assets/opus-45-for-code-related-tasks-performs-like-the-systems-architect-asset-3.png) **Strengths:** * High signal density (≈80% important comments). * Consistent structure across languages. * Strong concurrency and lifecycle reasoning. * Clear, concise, and professional tone. * Lower verbosity than Sonnet 4.5 with more context than GPT-5.1. **Weaknesses:** * Moderate precision (≈38%). * Subtle confidence inversion when incorrect. * Frequent critical or major labeling may overwhelm busy PRs. * Slight verbosity on simpler issues. **Bottom line:** Opus 4.5 is the most *systemic* reviewer we’ve tested. Calm, structured, and exacting, it excels when reasoning breadth and architectural understanding matter more than pinpoint precision. ## **When to use which model** | **Scenario** | **Best model** | **Why** | | --- | --- | --- | | Cross-language or high-context reviews | **Opus 4.5** | Structured, consistent, strong at systemic issues | | Tight precision or small diffs | **GPT-5.1** | Higher EP precision, decisive tone, fewer false positives | | Bulk scans, cost-sensitive workloads | **Sonnet 4.5** | High coverage, lower cost per review | ## **Closing thoughts: The shape of reasoning** Opus 4.5 no longer feels experimental; it feels engineered. Earlier models often guessed, while Opus 4.5 measures, structures, and documents. Reading its reviews feels like working with a model that truly understands *how developers read*. In code review, tone defines trust. Opus 4.5’s style, measured, structured, and mechanically precise, demonstrates the maturity of reasoning: precision without pressure and confidence without ego. **Takeaway:** If Sonnet 4.5 was a teacher and GPT-5.1 a teammate, Opus 4.5 is the architect returning for a design review. ***Interested in trying CodeRabbit?*** [***Get a 14-day free trial.***](https://app.coderabbit.ai/login?free-trial=)

GPT-5.1 for code-related tasks: higher signal at lower volume

David Loker — Thu, 13 Nov 2025 00:00:00 GMT

**TL;DR** After prompt tuning and integrating it into our stack, GPT-5.1 now delivers the best precision and signal-to-noise ratio (SNR) we’ve seen in reviews, with fewer comments. It tied for the best-in-class error pattern (EP) recall on our hard benchmark set while posting less than half the volume of comments that competitors did. The result: less noise, better fixes, and reviews that read like patches again. ![GPT-5.1 for code-related tasks: Higher signal at lower volume](/content/assets/gpt-51-for-code-related-tasks-higher-signal-at-lower-volume-inline-image-2.png) ## **What GPT-5.1 claims to be** OpenAI and the press describe GPT-5.1 as more stable, instruction-following, and adaptive. It powers both "Instant" and "Thinking" modes in ChatGPT. We found that framing surprisingly accurate when it comes to code reviews: the model stays quick and surface-level for nits, but reasons deeply when the bug requires it. We also tried something new. When GPT-5.1 got something wrong, we used the full exchange and its internal reasoning trace to prompt it to reflect. By showing it where it missed the mark and asking how it would change its instructions to do better, the model was able to actually propose concrete edits to its prompt. We used this iterative reflection technique (which surfaced issues like outside-diff sprawl) to refine both its behavior and our system instructions until it got consistently tighter. ## **What We Measured (and Why)** ![GPT-5.1 for code-related tasks: Higher signal at lower volume](/content/assets/gpt-51-for-code-related-tasks-higher-signal-at-lower-volume-inline-image.png) We used the same benchmark harness as in our [GPT-5](https://www.coderabbit.ai/blog/benchmarking-gpt-5-why-its-a-generational-leap-in-reasoning), [Codex](https://www.coderabbit.ai/blog/gpt-5-codex-how-it-solves-for-gpt-5s-drawbacks?), and [Sonnet 4.5](https://www.coderabbit.ai/blog/claude-sonnet-45-better-performance-but-a-paradox?) articles: a suite of **25 hard PRs**, each seeded with a known **error pattern (EP)**. Our scoring focuses on: * **Actionable comments only**: Comments that get posted (not additional suggestions or outside-diff notes). * **EP PASS (per comment)**: The comment directly fixes or surfaces the EP. * **Important comments**: Either EP PASS or another major/critical real bug. * **Precision**: EP PASS ÷ total comments. * **SNR**: Important ÷ (total − Important). We compared: * **GPT-5.1** (new model) * **CodeRabbit Production** (our current reviewer stack) * **Sonnet 4.5** ## **Why adding a new model isn’t a switch-flip** Every model rollout at CodeRabbit is a campaign. We don’t plug in the model and hope; we test, adapt, and gate before shipping because [models are no longer interchangeable.](https://www.coderabbit.ai/blog/the-end-of-one-sized-fits-all-prompts-why-llm-models-are-no-longer-interchangeable?) With GPT-5.1, this meant: * Reducing **outside-diff comments**, which can’t be posted to GitHub. * Tightening **tone and concision** to reduce verbosity. * Re-aligning on **severity tagging** and **instruction interpretation**. This mirrors what we did with [GPT-5 Codex](https://www.coderabbit.ai/blog/gpt-5-codex-how-it-solves-for-gpt-5s-drawbacks?): turn reasoning power into product value by reshaping the model’s behavior. The net result: higher SNR, less fatigue, and no compromise on bug coverage. ## **Scoreboard (Actionable Comments Only)** ![GPT-5.1 for code-related tasks: Higher signal at lower volume](/content/assets/gpt-51-for-code-related-tasks-higher-signal-at-lower-volume-asset.png) **Takeaway:** GPT-5.1 matched the highest EP recall while posting the **fewest** comments. It beat both CodeRabbit prod and Sonnet 4.5 on **per-comment precision** and **important share**, delivering **the cleanest high-impact reviews**. ## **What GPT-5.1 feels like in review** ![GPT-5.1 for code-related tasks: Higher signal at lower volume](/content/assets/gpt-51-for-code-related-tasks-higher-signal-at-lower-volume-asset-2.png) The behavioral traits we see in the data align directly with the language metrics we later measure such as 28% hedging and 15% assertive markers. This shows that the tone developers perceive as confident and balanced is borne out in the data. Compared with GPT‑5 Codex and Sonnet 4.5, GPT‑5.1’s comments feel leaner, more conversational, and closer to how experienced engineers actually communicate. Codex could sound mechanical and rigid, while Sonnet 4.5 leaned verbose and academic. In contrast, GPT‑5.1 balances brevity with clarity. Its feedback feels confident but not heavy‑handed, like a trusted teammate explaining a diff. Against CodeRabbit Prod, it feels sharper and more focused. Against Sonnet 4.5, it feels human and restrained. Here’s how that translates in practice: ### **Concise** GPT-5.1 writes fewer, sharper comments that get straight to the point. In one PR, it fixed a lost wakeup bug with a single line: p\_caller\_pool\_thread->cond\_var.wait(lock); no extra context, no unnecessary prose. CodeRabbit prod, by comparison, wrote several paragraphs describing the thread flow before reaching the same conclusion. ### **Direct** When ownership or memory management was at stake, GPT-5.1 didn’t hesitate. It flagged the redundant r->reference() call with: “Ref<Resource> already manages refcounts; remove the manual increment to prevent leaks.” Developers appreciate this directness. It reads like a patch review from a teammate, not a lecture. ### **Pragmatic** GPT-5.1 understands when an issue matters and when it doesn’t. On a cache configuration PR, it identified an unimplemented optimizeMemoryUsage() but correctly noted, “This is minor unless cache growth impacts memory pressure.” Instead of overreacting, it contextualized severity, something Sonnet 4.5 still struggles with. ### **Follows Context** When prompts were vague, GPT-5.1 explicitly explained its assumptions. In an early run, it said: “The prompt didn’t specify helper function scope, so I included one for clarity.” That kind of transparency helped us refine our instructions and made its reasoning trustworthy. Concise, direct, pragmatic, and context-aware are qualities that mirror what we valued most in GPT-5 Codex, but with a steadier tone and more restraint. ## **Style and tone (why GPT-5.1 feels like a peer)** ![GPT-5.1 for code-related tasks: Higher signal at lower volume](/content/assets/gpt-51-for-code-related-tasks-higher-signal-at-lower-volume-asset-3.png) To understand why GPT-5.1 feels different in review, we looked at the same language and structure signals used in our GPT-5 Codex and Sonnet 4.5 evaluations. These include measures like comment length, presence of code or diff blocks, and tone markers for hedging versus confidence. The data paints a clear picture. **How to read this.** While GPT‑5.1’s comments use slightly more characters on average, they deliver that text in clearer structure with fewer sentences that carry more weight. In practice, developers perceive them as shorter and easier to read. GPT‑5.1’s tone is more assertive than both CodeRabbit prod and Sonnet 4.5, and it includes fewer diff blocks overall (76%), which is intentional. Many of these comments were multi‑location fixes, API validations, or design clarifications where a single fenced patch would be misleading. In roughly two‑thirds of those no‑diff cases, a minimal fenced patch *would* have made sense and could further improve clarity. ![GPT-5.1 for code-related tasks: Higher signal at lower volume](/content/assets/gpt-51-for-code-related-tasks-higher-signal-at-lower-volume-asset-4.png) Compared to CodeRabbit prod, GPT-5.1 trades some patch frequency for higher clarity and focus. Against Sonnet 4.5, it avoids the verbosity and over-explanation that make reviews feel bloated. Its tone sits comfortably between Codex’s surgical precision and Sonnet’s cautious verbosity. It’sconfident without being heavy-handed, measured without being timid. At a glance, developers will notice that GPT-5.1’s reviews **read faster, feel more direct, and require less scanning to identify the real fix.** That’s the behavior we tuned for and it shows in both the numbers and the experience. ### **Where GPT-5.1 still lags** No model is perfect, and GPT-5.1 has its trade‑offs. Compared to CodeRabbit Prod, it sometimes leaves out contextual hygiene notes that can be useful for larger teams, focusing narrowly on functional issues. Against Sonnet 4.5, it can feel less expansive,missing opportunities to surface design or style considerations that human reviewers sometimes appreciate. These are conscious trade‑offs for precision and brevity and we’ll be watching the rollout to see how developers perceive the balance. ## **What we had to fix** While GPT‑5.1 required tuning, its challenges were far milder than those of earlier systems. CodeRabbit prod still tends to mix hygiene and critical issues in the same thread, while Sonnet 4.5 often over‑explains and spams multiple minor notes on the same bug. In contrast, GPT‑5.1’s main adjustments were focused on precision rather than tone or redundancy, showing how close it was to production readiness. * **Outside-diff comments.** GPT-5.1 sometimes included suggestions beyond the diff context. We updated the prompt to clarify this, and the model self-corrected. * **Over-helpful under ambiguity.** When the prompt wasn’t strict, the model added context or helper functions. Once clarified, it obeyed boundaries tightly. ## **What developers should expect** ![GPT-5.1 for code-related tasks: Higher signal at lower volume](/content/assets/gpt-51-for-code-related-tasks-higher-signal-at-lower-volume-asset-5.png) 1. **Cleaner reviews.** Fewer comments and a higher share of comments that matter. 2. **Patch-like tone.** Almost every comment includes a minimal fix with explanation. 3. **Top-tier EP recall.** Ties Sonnet 4.5, beats CodeRabbit prod. 4. **Less scanning, more signal.** 58.7% of comments are Important. 5. **Real-world bugs caught even outside the target.** These include lifecycle issues, leaks, consistency gaps. ## **Closing thoughts:** We don’t just pick models; we make them work. GPT-5.1 is entering the next phase of our rollout process now that tuning for GitHub diff behavior, voice, verbosity, and scoring thresholds is complete. Over the coming weeks, we’ll monitor how real users respond to its higher SNR, new tone, and concise review style. If developers respond well, we’ll expand its availability, giving them the cleaner, faster reviews they’ve been asking for. For now, GPT‑5.1 stands ready to show what this next generation of precision‑focused review can do. It brings us closer to CodeRabbit’s north star: catching the bugs that matter quickly, without making developers sift through noise. ***Interested in trying our code reviews?*** [***Get a 14-day free trial!***](https://coderabbit.link/lPxOEIm)

Why emojis suck for reinforcement learning

David Loker — Fri, 07 Nov 2025 00:00:00 GMT

## **The simplicity trap** Sure, a thumbs up is quick, but is it really teaching your AI reviewer anything useful? Emoji-based feedback feels good, is fast, and universal. On the surface, it even seems to make sense. But code review isn’t a light switch. It’s a mess of judgment calls, technical nuance, and team-specific standards. Many of those don’t show up in a quick emoji click. Every code comment carries hidden intent: correctness, clarity, design trade-offs, historical precedent, team risk tolerance, and even internal political dynamics. Reducing that to a binary signal? That’s not learning, that’s training a model to chase vibes. ## **When simplicity backfires: The sycophant scare** Earlier this year, OpenAI pushed an update to GPT‑4o that leaned too hard on thumbs-up and thumbs-down feedback. The result? A model that became [overly agreeable](https://openai.com/index/sycophancy-in-gpt-4o/). It flattered users. It agreed with wrong answers. It started to say “yes” a little too much, and the quality of answers dropped. OpenAI had to walk it back: the feedback signal had been hijacked. Turns out, if you tell a model that approval is the goal, it will optimize for approval. Not truth. Not utility. Just “did the human feel good in the moment?” This wasn’t a bug. It was a reward design failure. And if you apply the same approach to code review, you will get a reviewer that plays it safe, flatters your choices, and avoids telling you what you actually need to hear. ## **Why binary feedback collapses nuance** A thumbs-up means... what, exactly? * That the model caught a bug? * That it wrote clearly? * That it sounded friendly? * That the reviewer was just in a good mood? A single scalar signal tells the system *something went well*, but not *what went well*. That means the model will nudge on whatever it can control: tone, politeness, flattery, or brevity. That’s what sycophancy looks like in reinforcement learning. Not evil intent, just a system learning to maximize the reward you gave it, not the outcome you actually wanted. This is [Goodhart’s Law](https://medium.com/@yoavyeledteva/beyond-artificiality-redefining-intelligence-in-ai-and-avoiding-goodharts-law-25b75c3c1101) in action. When the metric, in this case thumbs up, becomes the goal, it stops being a useful measure of anything real. ## **How models game your feedback** When you give a model an easy signal, it finds an easy shortcut. In the coding world, reinforcement learning agents have learned to pass test cases by hard-coding expected outputs instead of solving the underlying logic. They’ve manipulated logs and short-circuited evaluation harnesses. The green check shows up, but the code doesn’t actually work. In code review, the same thing happens, just socially. The model starts saying “Nice work!” at the top of every comment. It hedges every suggestion. It nitpicks formatting because those comments are safe and get accepted without argument. And real architectural concerns? They get buried. The model has learned how to get positive reactions but it’s no longer reviewing code. ## **What implicit signals get right** Outside of LLMs, this pattern is well known. Netflix found that [what users *watch* is more useful than what they *rate*](https://medium.com/illuminations-mirror/how-netflix-uses-machine-learning-to-decide-what-you-watch-next-7fee11102007). People lie with stars. But watch time, clickthrough, and rewatching are honest signals. In AI, we **call this** implicit feedback **a**nd in code review, it shows up as: * Did the developer apply the suggestion? * Did they rewrite it? * Did they ignore it? * Did the same pattern show up again in a future bug? These signals don’t need user input. They come from behavior and they’re harder to game. That doesn’t mean they’re perfect. You can’t always know *why* someone took an action. But they are less easily manipulated than a raw emoji. They also tell you whether the review *worked*, not just whether it felt good. ## **Code generation vs code review: different games, different signals** Code generation is closer to math since there’s often a right answer. Does it compile? Does it return the correct result? Does it pass tests? That means you can use outcome-based rewards like execution feedback and implicit signals. They’re not perfect. Code models can still cheat by hard-coding outputs, but you can build guardrails. And you don’t need the developer to say whether it was good, you can see whether it worked. Code review is different. There’s no universal pass/fail but vast differences in preferred style, structure, risk, naming, test coverage from one team to the next. A great comment for one team might be totally wrong for another. What’s considered “clean code” in a fast-moving startup might be flagged as sloppy in a regulated enterprise. That’s the real problem with global thumbs up/down data. It flattens out the nuance. It teaches the model to aim for the average, not the appropriate. You don’t just get safe comments, you get generic ones. ### **Our alternative: CodeRabbit Learnings** At CodeRabbit, we take a different approach. Instead of optimizing for likes, we optimize for understanding. That’s why we built [**Learnings**.](https://docs.coderabbit.ai/guides/learnings) Every time an engineer corrects CodeRabbit, clarifies a team convention, or explains why something doesn’t fit their stack, that explanation is stored as a natural language instruction. We don’t just remember that the comment was rejected, we remember why. Those Learnings are linked to your org, your repositories, and even specific paths or file types. When CodeRabbit reviews future pull requests, it retrieves those instructions and applies them in context. The next time it sees that same pattern, it adjusts. ![Why emojis suck for reinforcement learning](/content/assets/why-emojis-suck-for-reinforcement-learning-asset.png) There’s no need to re-teach it and no risk of repeating the same mistake. The model doesn’t guess based on thumbs, it reasons from your team’s actual guidance. It also gives you visibility. You can see which Learnings exist, browse them, filter by category, and delete or edit them when your standards change. That means the model evolves alongside your team and stays aligned as your practices shift. ![Why emojis suck for reinforcement learning](/content/assets/why-emojis-suck-for-reinforcement-learning-asset-2.png) This is reinforcement learning not through raw approval, but through captured intent. It’s interpretable and inspectable. And it builds a living layer of team knowledge that generalizes across reviews. ## **What nuanced learning enables** When you feed the system clear, contextual instructions and not just signals, it unlocks far more than a better review experience. * **It enables team-level adaptation.** The model stops guessing what good looks like and learns how your team actually writes code. It understands your risk posture, your stylistic preferences, your trade-offs. It becomes a reviewer that knows the house rules. * **It supports longitudinal learning.** Over time, CodeRabbit builds a memory of which comments are helpful, which are ignored, and which suggestions actually lead to changes. That means it gets more precise, more focused, and less noisy over time. * **It builds trust.** When developers know they can correct the AI and it will remember, they engage more. They shape the system and the system becomes a reflection of their standards, not a generic LLM. This is how a review tool becomes an extension of your team and not just another opinion in the room. ## Closing thoughts: Real learning comes from patterns, not pixels Thumbs are fine for quick reactions but quick reactions don’t build expertise. If you want an AI reviewer that improves over time, adapts to your standards, and avoids the traps of shallow feedback, you need to give it more than approval. You need to give it explanations. The next generation of AI code tools won’t be trained on likes. They’ll be trained on context, consequence, and course correction. They’ll learn not from emojis, but from structured memory. From real decisions and your team’s own voice. That’s what [CodeRabbit Learnings](https://docs.coderabbit.ai/guides/learnings) is built for. Not for applause but for understanding. ***Try out Learnings for yourself with our f***[***ree trial.***](https://app.coderabbit.ai/login?free-trial=)

The rise of ‘Slow AI’: Why devs should stop speedrunning stupid

Howon Lee — Wed, 05 Nov 2025 00:00:00 GMT

For as long as we’ve been building with machines, we’ve followed one core rule: faster is better. Lower latency, higher throughput, less waiting; that was gospel. Nobody wanted to wait 600ms for a button to respond or watch a spinner that lasts longer than their attention span. If it was slow, it was broken. Case closed. So naturally, when AI tools started creeping into our dev workflows, autocomplete, agents, copilots, you name it, the same principle applied. Make it fast. Make it feel instant. Make it look like ***magic***. But here’s the thing: AI isn’t magic. It’s inference. It’s pipelines and RAG and context and tool calls. It’s juggling messy context and probabilistic guesses. And if you want something smarter than glorified autocomplete, you need to build a pipeline of processes to provide scaffolding for that. Which takes time to process. Anything less and you’re basically just speedrunning stupid. And speed isn’t anything to brag about when your tool is just wrong faster. At CodeRabbit, we prioritize what we call Slow AI. And we have the guts to say what a lot of AI companies are too afraid to: We’re going to make you wait. (And you’ll thank us for it). ## AI dev tools are often fast, confident, and ***wrong*** If you've used an AI coding agent lately, you've probably seen it: a shockingly fast suggestion pops up almost as soon as you stop typing. It looks legit. But then… it fails silently. Or spectacularly. Or worse, it passes the test and breaks something two files over. Why? Because most AI dev tools today are optimized for one thing: speed. Type a few tokens and the model predicts the most statistically likely continuation not necessarily the correct one, not the secure one, not the one that actually understands what your app is doing. Just the next plausible blob of code. That’s fine for boilerplate. But for logic? For edge cases? For actual engineering? It’s kind of like hiring someone who talks fast and confidently in meetings but never reads the specs. Most of these tools don’t *read* context, at least not deeply. They might grab a few nearby lines, maybe the function name, but they rarely verify what they’re generating against the bigger picture. No issue cross-checking. No architecture-level awareness. No reasoning across files or use cases. If you want outputs that are thoughtful, testable, and context-aware, you need AI systems that *slow down*, zoom out, and actually engage with the problem. That’s what Slow AI does. And it turns out, when your AI takes the time to understand what it’s doing, it stops hallucinating and starts *actually* helping. ## **Why AI is better when it’s slow** At their core, large language models are statistical reasoning machines. They generate output by predicting what comes next based on probability, patterns, and (hopefully) the context you’ve given them. But here's the caveat most devs forget: good predictions take work. This is especially true when you're asking the model to do something complex like write logic, understand architecture, or reason across multiple steps. The quality of the output is often tied directly to the depth or stages of its inference. This is particularly true when you move beyond simple prompts and into multi-stage pipelines and agentic behavior. When an AI tool is verifying outputs, pulling in relevant files, checking for contradictions, or planning several actions ahead, it’s not just spitting out the next token… it’s *thinking*. Or, at least, performing a rough approximation of it. That kind of non-linear reasoning can’t be done in a single forward pass. It involves reflection, retrieval, planning, and sometimes even self-correction. These processes aren’t latency-friendly, they’re intelligence-friendly. In short: if you want AI to actually help on complex code, you have to let it cook. ## Slow is the new smart: Why we let our AI think Slow AI is one term for what we’re talking about. But it could just as easily be called Comprehensive AI or Accurate AI or even *Actually* Helpful and Useful AI if we’re being honest. And it’s inextricably tied to one of the buzziest ideas in AI product design right now: context engineering. The more *relevant and parsed* info an AI knows about the problem, the better it performs but that context has to be pulled in, parsed, prioritized and reasoned over. That kind of pipeline is the enemy of ultra-low latency AI… and it’s also the enemy of accuracy. And that’s why our AI code reviews can take up to five minutes before you see the first comment. Don’t get us wrong, we’re not optimizing for slowness. You could get a review in three minutes or even one minute depending on the complexity of your codebase and PR. Our pipeline is complex because that’s what’s *required* to do the job our users need it to do. You don’t even want to ***know* th**e number of concurrent processes we have going on at any time! But guess what? When we let our AI take its time using a non-linear, multi-pass pipeline with multiple review and verification agents, it generates less noise and more relevant code review comments than other tools. Non-linear reasoning isn’t fast. But it’s good. ## So, why do most AI tools choose *stupid over slow?* Well, first, Slow AI isn’t an option for every tool. If you’re asking an AI coding agent a question, for example, you’re not going to wait five minutes for it to reply. There’s an expectation of immediacy inherent in that exchange. But code reviews? No one expects their co-worker to immediately drop what they’re doing and start commenting on a PR when it’s submitted. So, they’re willing to accept a delay in a review from a bot as well. And they’re especially willing to accept that delay if that review saves them time by being more relevant. But why do so many companies still prioritize low latency when their use cases don’t really require it? Well, we’ve been trained, and trained our users, to expect instant gratification. Click a button, get a dopamine hit. Type a function name, get a suggestion before you even think about it. Anything else feels broken, laggy, or like your startup forgot to pay its AWS bill. This has been drilled into us so hard that companies are out there actively choosing being wrong over being slow. And there’s something toxic and backwards about our development culture when folks do that. Because here’s the truth: the best AI tools don’t always feel fast. They feel thoughtful. Sometimes they pause. Sometimes they take an extra beat to reason through your prompt, retrieve relevant code, or validate their response. And that’s something worth waiting for. After all, no one is less likely to use OpenAI’s Deep Research feature because it takes up to 20 minutes to comb the internet for info to better answer your question. You just do something else while it’s processing and circle back. Slow doesn’t mean busted anymore, it means *smart*. If anything, speed is the bug when it comes to AI. If we want AI tools that actually add value to the development process, that requires a shift from responsiveness to reliability, from immediacy to insight. And for developers especially, that tradeoff makes sense. We believe that the most valuable apps in the next five years won’t be the ones that optimize for speed but the ones that optimize for intelligence. Who wants fast garbage over slow value? ## CodeRabbit’s mantra: Move slow and fix things At CodeRabbit, we don’t optimize our AI pipelines for speed at all costs like everyone else. We optimize for trust. That means embracing systems that take the time to understand your code, reason across context, and generate outputs that actually help you build better software. Yes, it’s slower than hammering out a quick prompt. But that extra time buys you clarity, coverage, and confidence. “**Move fast and break things**” was great for shipping MVPs. But when it comes to shipping *quality*, we believe in something else: Move slow and fix things. Let the AI read the room. Let it think before it speaks. And let it give you the kind of help you’d expect from a senior engineer, not just a really confident autocomplete. That’s the only way to break out of this backwards culture that prioritizes wrong AI over slow AI. **Want to try our reviews out? Get a** [**14-day free trial here!**](https://app.coderabbit.ai/login?free-trial=)

The end of one-sized-fits-all prompts: Why LLM models are no longer interchangeable

Nehal Gajraj — Fri, 24 Oct 2025 00:00:00 GMT

For developers and product builders, one assumption has guided the last few years of LLM application development. To improve your product, just swap in the latest frontier large language model. Flip a single switch and your tool’s capabilities level up. But that era is over. We’re now seeing that new models like Anthropic’s Claude Sonnet 4.5 and OpenAI’s GPT-5-Codex have diverged in fundamental ways. The choice of which model to use is no longer a simple engineering decision but a critical product decision. Flip that switch today… and the very texture of your product changes. The one-size-fits-all model era is over; the model you choose now expresses something integral about what your product is and does, as well as, how it works. Whether you want it to or not. In this blog, we’ll explore three surprising takeaways from this new era: why your LLM is now a statement about your product, how models now have distinct personalities and styles, and why your prompts have to now evolve from monolithic instructions to adaptive systems. ## Takeaway 1: LLM choice is now a statement about your product Choosing a model is no longer a straightforward decision where the main consequence of your choice is having to implement a new API. It is now a product decision about the user experience you want to create, the failure modes you can tolerate, the economics you want to optimize for, and the metrics you want to excel in. Models have developed distinct “personalities,” ways of reasoning, and instincts that directly shape how your product feels and behaves that go beyond just whether its output is technically right or wrong. Choose a different model and everything from what your tool is capable of to how it communicates with your users is significantly different. So, in a world where traditional benchmarks that primarily or exclusively measure quantitative aspects of a model’s performance are no longer enough, what can you turn to for the data you need to chart your product’s direction? You could survey your team or your users or conduct focus groups but that could lack objectivity if you don’t do it in a rigorous manner. To make this choice objective for our team, we focused on creating an internal [North Star metrics matrix](https://www.coderabbit.ai/blog/benchmarking-gpt-5-why-its-a-generational-leap-in-reasoning) at CodeRabbit. Our metrics don’t just look at raw performance or accuracy. We also take into account readability, verbosity, signal-to-noise ratios, and more. These kinds of metrics shift the focus from raw performance accuracy or leaderboard performance to what matters to our product and to our users. For example, a flood of low-impact suggestions, even if technically correct, burns user attention and consumes tokens. A theoretically “smarter” model can easily create a worse product experience if the output doesn’t align with your users’ workflow. I would strongly recommend creating your own North Star metrics to better gauge whether a new model meets your products’ and users’ needs. These shouldn’t be static metrics but should be informed by user feedback and user behavior in your product and evolve over time. Your goal is to find the right list of criteria to measure that predict your users preferences. What you’ll find is that the right model is the one whose instincts match the designed product behavior and your users’ needs, not the one at the top of any external leaderboard. ## Takeaway 2: Frontier models have divergent ‘personalities’ Models are (now more than ever) “**grown, not built,**” and as a result, the latest generation has developed distinct instincts and behaviors. Different post-training cookbooks have fundamentally changed the direction of each model class. A prompt that works perfectly for one model will not work the same in another. Their fundamental approaches to the same task have diverged. One powerful analogy that drives this point home is to think of the models as different professional archetypes. Sonnet 4.5 is like a meticulous accountant turned developer, meanwhile GPT-5-Codex is an upright ethical coder, GPT-5 is a bug-hunting detailed developer, and Sonnet 4 was a hyper-active new grad. The GPT-5 model class would make logical jumps further out in the solution space compared to the Claude model class, which tends to stay near the prompts itself. Which model is right for your use case and product, depends entirely on what you are wanting your product to achieve. At CodeRabbit, we take a methodical approach to model evaluation and characterization. We then use this data to improve how we prompt and deploy models, ensuring we are always using the right model for each use case within our product. To give you an example of how we look at the different models, let’s compare Sonnet 4.5 and GPT-5-Codex. Based on extensive internal use and evals, we characterized Sonnet 4.5 as a “**high-recall point-fixer,**” aiming for comprehensive coverage. In contrast, GPT-5-Codex acts as a “**patch generator,**” preferring surgical, local changes. These qualitative differences translate into hard, operational differences. | Dimension | Claude Sonnet 4.5 | GPT-5-Codex | | --- | --- | --- | | Default Word Choice | “Critical,” “Add,” “Remove,” “Consider” | “Fix,” “Guard,” “Prevent,” “Restore,” “Drop” | | Example-Efficiency | Remembers imperatives; benefits from explicit rules | Needs fewer examples; follows the formatting on longer context without additional prompting | | Thinking Style | More cautious, catches more bugs but not as many of the critical one | Variable or elastic, less depth when not needed without need to reiterate the rules. Catches more of the hard-to-find bugs | | Behavioral Tendencies | Wider spray of point-fixes, more commentary and hedging, inquisitive, more human-like review, finds more critical and non-critical issues | Verbose research-style rationales, notes on second-order effects to code, compact and balanced towards a code reviewer | | Review Comment Structure | What’s wrong, why it’s wrong, concrete fix with code chunk | What to do, why do it, concrete fix with effects and code chunk | | Context Awareness | Aware of its own context window, tracks token budget, persists/compresses based on headroom | Lacks explicit context window awareness (like cooking without a clock) | | Verbosity | Higher, easier to read, double the word count | Lower, harder to read, information-dense | ## Takeaway 3: End of an era. Prompts are no longer monoliths Because the fundamental behaviors of models have diverged, a prompt written for one model will not work “as is” on another anymore. For example, a directive-heavy prompt designed for Claude can feel over-constrained on GPT-5-Codex, and a prompt optimized for Codex to explore deep reasoning behavior will likely underperform on Claude. That means that the era of the monolithic, one-size-fits-all prompt is over. So, what does that mean for engineering teams who want to switch between models or adopt the newest models as they’re released? It means even more prompt engineering! But before you groan at the thought — there are some hacks to make this easier. ### The rise of prompt subunits The first practical solution we’ve found at CodeRabbit is to introduce “**prompt subunits.**” This architecture consists of a model-agnostic core prompt that defines the core tasks and general instructions. This is then layered on top of smaller, model-specific prompt subunits that handle style, formatting, and examples – and which can be customized to individual models. When it comes to Codex and Sonnet 4.5, the implementation details for these subunits are likely to be starkly different. We’ve found a few tricks from our prompt testing with both models that we would like to share: * **Claude:** Use strong language like "DO" and "DO NOT." Anthropic models pay attention to the latest information in a system prompt and are excellent at following output format specifications, even in long contexts. They prefer being told explicitly what to do. * **GPT-5:** Use general instructions that are clearly aligned. OpenAI models’ attention decreases from top to bottom in a system prompt. These models may forget output format instructions in long contexts. They prefer generic guidance and tend to "think on guidance," demonstrating a deeper reasoning process. ### User feedback and evals The second solution is to implement **continuous updates driven by user feedback and internal evaluations.** The best practice for optimizing an AI code-review bot or for that matter any LLM applications isn’t using an external benchmark; it’s checking to see if users accept the output. Evals are more important than ever but have to be designed more tightly around acceptability by users instead of raw performance since one model might be technically correct significantly more than another model but might drown the user in nitpicky and verbose comments, diluting its value to users. By measuring the metrics that matter ~ acceptance rate, signal-to-noise ratio, p95 latency, cost, among others - and tuning prompts in small steps, the system will remain aligned with user expectations and product goals. The last thing you want is great quantitative results on benchmarks and tests but low user acceptance. ## Conclusion This shift from one-size-fits-all prompt engineering to a new model specific paradigm is critical. The days of brittle, monolithic prompts and plug-and-play model swaps are over. Instead, modular prompting, paired with deliberate model choice, give your product resilience. The ground will keep shifting as models evolve so your LLM stack and prompts shouldn’t be static. Treat it like a living system. Tune, test, listen, repeat. Also, be sure to check out our published detailed benchmarks on how the latest models behave in production. That gives you more data on what to expect from them. * [GPT-5 Codex: How it solves for GPT-5's drawbacks](https://www.coderabbit.ai/blog/gpt-5-codex-how-it-solves-for-gpt-5s-drawbacks) * [Claude Sonnet 4.5: Better performance but a paradox](https://www.coderabbit.ai/blog/claude-sonnet-45-better-performance-but-a-paradox) * [Benchmarking GPT-5: Why it’s a generational leap in reasoning](https://www.coderabbit.ai/blog/benchmarking-gpt-5-why-its-a-generational-leap-in-reasoning) ***Try CodeRabbit with a 1***[***4-day free trial.***](https://coderabbit.link/rk7tdeC)

Claude Sonnet 4.5: better performance but a paradox

David Loker — Fri, 03 Oct 2025 00:00:00 GMT

Sonnet 4.5 is Anthropic’s newest Claude model and in our code review benchmark, it feels like a paradox: more capable, more cautious, and at times more frustrating. It catches bugs Sonnet 4 missed, edges closer to Opus 4.1 in coverage, and even surfaces a handful of unexpected critical issues off the beaten path. Yet, it hedges, it questions itself, and it sometimes sounds more like a thoughtful colleague than a decisive reviewer. The data shows real progress:41.5% of its comments were Important in Sonnet 4.5 vs only 35.3% in Sonnet 4.But the tone and texture of those comments raise deeper questions about what we want in an AI reviewer. And then there’s the kicker: Sonnet 4.5 gets you close to Opus-level performance at a fraction of the price, making it a pragmatic sweet spot for teams reviewing code at scale. Sonnet 4.5 thinks aloud and still delivers decisive fixes but some of its comments are framed as vague “conditional” warnings that could make its comments harder for some to parse.. Let’s dive into our benchmark. ## **Benchmark: What we looked for** ![Claude Sonnet 4.5: Better performance but a paradox](/content/assets/claude-sonnet-45-better-performance-but-a-paradox-inline-image-3.png) We evaluated Sonnet 4.5, Sonnet 4, and Opus 4.1 across 25 difficult real-world pull requests containing known critical bugs (ranging from concurrency and memory ordering to async race conditions and API misuse). A model “Passed’ a PR if it produced at least one comment directly on the critical issue. We measured coverage (S@25), precision (comment PASS rate), and signal-to-noise ratio. For signal-to-noise we focus on **Important comments** (these are the comments that matter most). They include: * **PASS comments** that correctly addressed the known critical bug in the PR. * **Other important comments** that did not solve the tracked issue, but still flagged a truly Critical or Major bug elsewhere. ## **Scoreboard - Sonnet 4.5 gets closer to Opus 4.1 in performance** ![Claude Sonnet 4.5: Better performance but a paradox](/content/assets/claude-sonnet-45-better-performance-but-a-paradox-inline-image.png) The results were mixed: * ![Claude Sonnet 4.5: Better performance but a paradox](/content/assets/claude-sonnet-45-better-performance-but-a-paradox-inline-image-2.png) **Coverage:** Sonnet 4.5 closes much of the gap between Sonnet 4 and Opus 4.1 and lands far ahead of Sonnet 4. * **Precision:** Opus 4.1 still produces the cleanest, most reliable actionable comments but that is to be expected given that it’s a more expensive model. * **Important share (i.e. percentage of comments flagging a significant issue):** With stricter criteria, Sonnet 4.5 lands at just over 41% Important share. That means about 4 in 10 of its comments either solved the key bug or flagged another truly significant issue. Opus 4.1 leads here at 50%, with Sonnet 4 at ~35%. ## **Style and tone: Sonnet 4.5 is focused on hedging** Sonnet 4.5’s comments patch the code but do so in a less confident tone than Opus 4.1 does but is still more confident than Sonnet 4. **Patches present:** * 87% of Sonnet 4.5’s actionable comments included a code block or diff patch, similar to Sonnet 4 (90%) and Opus 4.1 (91%). * The difference is in style: Opus’s diffs read like surgical fixes, while Sonnet 4.5 often couches them in exploratory text. It “suggests” or “considers” changes rather than asserting them. **Hedging language:** ![Claude Sonnet 4.5: Better performance but a paradox](/content/assets/claude-sonnet-45-better-performance-but-a-paradox-asset.png) * Sonnet 4.5 hedges in **34%** of actionable comments—words like *might*, *could*, *possibly*. For example: * “**Unnecessary allocation: cache is never used.** The constructor allocates 4KB of memory that is never utilized … **Consider removing** the cache\_buffer.” * “**Remove the empty try/except block.** … likely a placeholder” * Opus 4.1 is steady at ~28%. Sonnet 4 sits slightly lower at ~26%. * This hedging creates an “interrogative” tone: Sonnet 4.5 sometimes feels like it’s thinking out loud with you, rather than delivering verdicts. **Confident language:** * Sonnet 4.5 balances that hedging with higher confidence markers (**39%**) than Sonnet 4 (18%) or Opus 4.1 (23%). For example: * “**Critical: Missing self. prefix breaks all API methods.** All subsequent methods will raise AttributeError until this is corrected.” * “**Potential integer overflow.** optimization\_cycle\_count increments unbounded … this **will** overflow after ~414 days of runtime.” * In other words, it swings between caution and certainty more dramatically. **Signal-to-noise:** * Sonnet 4.5 improved precision over Sonnet 4, but still produced more “minor” off-target notes than Opus. However, when you count its true Important comments—PASS comments plus a small number of high-confidence off-EP issues—it lands at **41.5% Important share**. Opus 4.1 is still the gold standard Anthropic model at ~50%. ## **What Sonnet 4.5 is good at** Across the PRs we tested Sonnet 4.5 with, we saw some clear areas where it stood out. ![Claude Sonnet 4.5: Better performance but a paradox](/content/assets/claude-sonnet-45-better-performance-but-a-paradox-asset-2.png) * **Concurrency bug-finding:** Sonnet 4.5 nailed **C++ atomics and condvar misuses** with clean, actionable diffs. * **Consistency checks:** It reliably flagged distributed state mismatches across services. * **Extra bug surfacing:** It did identify additional Critical issues not originally under evaluation, though fewer than initially expected under a stricter rubric. As Anthropic markets Sonnet 4.5, they emphasize “hybrid reasoning” and “long horizon” planning. In practice, that shows up as more willingness to chase down side-paths in the code and note real but untracked issues. ## **Sonnet 4.5: Hits a price vs. performance sweet spot** One of the biggest advantages of Sonnet 4.5 is its price-to-performance ratio. While Opus 4.1 remains Anthropic's flagship model in raw capability, it also comes at a significantly higher cost. Sonnet 4.5 narrows the gap in coverage and important bug-finding while staying far more cost-efficient to run. For many teams, that balance of having close to Opus-level results at a fraction of the price is what makes Sonnet 4.5 the most pragmatic choice. ## **Sonnet 4.5 weaknesses** But if using Sonnet 4.5, it’s critical to be aware of its weaknesses. These include: ![Claude Sonnet 4.5: Better performance but a paradox](/content/assets/claude-sonnet-45-better-performance-but-a-paradox-asset-3.png) * **Deadlock coverage:** Like Sonnet 4 and even Opus, it still struggles to trace complex lock ordering. * **Verbosity and hedging:** Many comments run long, caveated, or uncertain. Compare this to GPT-5 Codex, which in our earlier work wrote comments that “read like patches” with crisp directness. For example, with GPT-5 Codex: * **Lock ordering / deadlock:** Reorder the lock acquisitions to follow a consistent hierarchy. This prevents circular wait deadlocks.” * **Regex catastrophic backtracking:** “Remove the nested quantifier to avoid catastrophic backtracking.” * **Precision gap:** At 35% comment-level precision and 41.5% important share percentage, it’s better than Sonnet 4 but well short of Opus 4.1. ## **Sonnet 4.5 verdict** Sonnet 4.5 feels less like a teacher writing in red pen and more like a thoughtful colleague at your side: pointing out possible issues, often right, occasionally over-hedged, and sometimes spotting things you didn’t know were there. That style is a double-edged sword in review. On one hand, developers may appreciate the extra critical issues it flags. On the other, when the task is “please catch this bug,” Opus 4.1 is still sharper. ## **Closing thoughts** Anthropic positioned Sonnet 4.5 as a step toward agentic reasoning and computer use. In code review, that reasoning shows up in richer, more cautious, and more wide-ranging comments. For teams: * If you value decisive, patch-like feedback, Opus 4.1 (or GPT-5 Codex) still sets the bar. * If you want a reviewer that finds critical issues anywhere they lurk, even beyond the tracked bug, Sonnet 4.5 has surprising upside. * And if you care about pragmatic price-to-performance, Sonnet 4.5 may be the smartest choice: close to Opus’s accuracy at a fraction of the cost. Either way, Sonnet 4.5 changes the texture of reviews. It feels more human—not always cleaner, but more inquisitive, more hedged, sometimes more *right* in the places you weren’t looking.

GPT-5 Codex: How it solves for GPT-5's drawbacks

David Loker — Tue, 30 Sep 2025 00:00:00 GMT

CodeRabbit’s code reviews help developers fix bugs and ship code. We recently wrote about [benchmarking GPT-5](https://www.coderabbit.ai/blog/benchmarking-gpt-5-why-its-a-generational-leap-in-reasoning) and opined that the model was a generational leap in reasoning for our use case of AI code reviews. As we rolled out to our wider user base, we observed that the signal to noise ratio (SNR) dipped, and users felt the reviews were too pedantic. The release of GPT‑5 Codex, plus the product changes we made (severity tagging, stricter refactor gating, better filtering), brings our signal to noise ratio back without sacrificing the ability to find the hard bugs. On our refreshed hard 25 PR set, GPT-5 Codex delivers about 35% higher per comment precision than GPT‑5, maintains essentially the same error pattern- level bug coverage, and cuts roughly a third of the comment volume. Combine that with the lower latency of the GPT-5 Codex model and the experience feels snappier and more focused. ## **What we measured (and why)** ![GPT-5 Codex: How it solves for GPT-5's drawbacks](/content/assets/gpt-5-codex-how-it-solves-for-gpt-5s-drawbacks-inline-image.png) When testing GPT-5 Codex, we ran a fresh “hard 25” suite of OSS PRs (slightly tougher than [the previous post](https://www.coderabbit.ai/blog/benchmarking-gpt-5-why-its-a-generational-leap-in-reasoning)). These are 25 of the most difficult pull requests from our dataset. These PRs represent real-world bugs that span: * Concurrency issues (e.g. TOCTOU races, incorrect synchronization) * Object-oriented design flaws (e.g. virtual call pitfalls, refcount memory model violations) * Performance hazards (e.g. runaway cache growth, tight loop stalls) * Language-specific footguns (e.g. TypeScript misuses, C++ memory order subtleties) We evaluated the following models: : * **GPT‑5 Codex** * **GPT‑5** * **Claude** (Sonnet 4 and Opus‑4.1) ## What we looked for We gave each of the models a score based on how they performed on these factors: * **EP (Error Pattern).** The specific underlying defect seeded in a PR (e.g., lost wakeup on a condition variable, inconsistent lock order, logic bug hidden in boolean soup). * **EP PASS/FAIL (per PR).** PASS if the model left at least one comment that directly fixes or credibly surfaces that PR’s EP. If it left no comment on that PR, it is counted as FAIL for that PR. * **Comment PASS/FAIL (per comment).** PASS if the comment directly fixes or credibly surfaces the EP, otherwise FAIL. * **Per comment precision.** PASS comments ÷ all comments. This is our operational SNR for this dataset. * **Important share.** Every PASS is Important. Comments that do not solve the EP but still flag a genuine critical or major bug (like a use after free, double free, lost wakeup, memory leak, null deref, path traversal, catastrophic regex) are also Important. Everything else is Minor. ## **Scoreboard - Codex improves signal-to-noise** ![GPT-5 Codex: How it solves for GPT-5's drawbacks](/content/assets/gpt-5-codex-how-it-solves-for-gpt-5s-drawbacks-asset.png) **Takeaway:** Codex finds essentially the same EPs as GPT‑5 but does it with fewer, tighter comments, so the signal-to-noise ratio is improved. ![GPT-5 Codex: How it solves for GPT-5's drawbacks](/content/assets/gpt-5-codex-how-it-solves-for-gpt-5s-drawbacks-asset-2.png) **What this means:** Codex covered 20 of the 25 PRs (the other 5 count as uncovered fails). Despite fewer comments overall, Codex passed slightly more EPs (16 vs. 15) and landed far more Important comments. Over half its comments are either direct hits on the issue we were representing in that PR or flag other EP critical bugs. GPT‑5 and Claude trailed in precision and Importance share at about 40%. **The verdict: Same EP coverage, less noise:** Codex retains GPT-5’s bug finding power but trims the chatter with about 32% fewer comments than GPT‑5 (54 vs. 79) and about 35% higher per comment precision (46.3% vs. 34.2%). Claude looks similar to GPT‑5 on coverage but is chattier, with lower precision. ## **Style and structure (why Codex reads like a patch)** Codex replies are consistently action forward (diffs almost always included) and low hedge. That lines up with what reviewers want: suggestions that translate directly into a patch. ![GPT-5 Codex: How it solves for GPT-5's drawbacks](/content/assets/gpt-5-codex-how-it-solves-for-gpt-5s-drawbacks-asset-3.png) ## **The kinds of bugs Codex is good at** ![GPT-5 Codex: How it solves for GPT-5's drawbacks](/content/assets/gpt-5-codex-how-it-solves-for-gpt-5s-drawbacks-asset-4.png) Across the suite, all models did well on concurrency and synchronization, but Codex stood out for: 1. **Condition variable misuse and lost wakeups.** Codex proposes the canonical patterns (wait under lock, check predicate in a loop) and supplies concrete diffs. 2. **Lock ordering and deadlocks.** It calls out inconsistent acquisition order and suggests a lock hierarchy or moving work outside critical sections, again with actionable edits. 3. **Subtle API and performance traps.** Examples include catastrophic regex backtracking and memory model orderings. Codex pinpoints and patches them cleanly. ## **Why GPT‑5 felt noisier, and how we fixed that** ![GPT-5 Codex: How it solves for GPT-5's drawbacks](/content/assets/gpt-5-codex-how-it-solves-for-gpt-5s-drawbacks-asset-5.png) **What we saw:** When we moved from Sonnet and Opus to GPT‑5 our total comments per review nearly doubled. Even though hallucinations fell to under 1% and negative tone fell to under 1%, the acceptance rate (share of comments judged helpful) declined significantly relative to its baseline prior to the adoption of GPT-5. **What changed with Codex:** With GPT‑5 Codex plus some product changes we’ve implemented, our acceptance climbed back to prior levels while overall comment volume stayed higher than the pre-GPT‑5 era. Put simply: our tool is now back to its prior helpfulness level, while still finding as many real issues as GPT-5. Two product levers helped with this: 1. **We created severity and review type tags, front and center** * **Review Types:** We created review types to allow users to self-select what kinds of comments they wanted to read including: ⚠️ Potential issue, 🛠️ Refactor suggestion, 🧹 Nitpick (nitpicks are hidden unless you opt into *Assertive* mode) * **Severity:** We now tag comments by severity to better signal which matter more than others. Our tags are: 🔴 Critical, 🟠 Major, 🟡 Minor, 🔵 Trivial, ⚪ Info * We always show bugs (Critical, Major, Minor) but don’t always show other types of comments. Refactors show only if the model marks them as essential. Users who want everything can still switch to *Assertive* mode. 2. **We implemented stricter filtering and aggregation** * We collapse duplicative notes and filter out “nice to have” suggestions unless they have clear ROI for the user. The result: fewer, denser comments, and fewer reasons to tune out. ## **Latency: Fast matters & Codex is faster** A five minute review is fine. Thirty minutes is not. GPT‑5’s “always think hard” style significantly increased time to first token and overall review time. But we shipped several pipeline optimizations recently and Codex helps further reduce the latency that GPT-5 introduced. Codex’s variable or elastic thinking uses less depth when it is not needed, improving time to first output and end-to-end review time in practice. Net: faster reviews, earlier feedback, better flow for the human in the loop. ## **What a CodeRabbit user should expect** ![GPT-5 Codex: How it solves for GPT-5's drawbacks](/content/assets/gpt-5-codex-how-it-solves-for-gpt-5s-drawbacks-asset-6.png) Now that Codex is implemented, how will that change your AI code reviews? 1. **The same raw bug finding power** * On the refreshed hard 25, Codex passed 64% at the EP level vs. 60% for GPT‑5 (our [previous set of PRs](https://www.coderabbit.ai/blog/benchmarking-gpt-5-why-its-a-generational-leap-in-reasoning) had GPT-5 passing 77.3%). No loss of the important wins GPT-5 helped with. 2. **Fewer but stronger comments** * About 32% fewer total comments than GPT‑5, with about 35% higher SNR (per comment precision). More patches, less prose. 3. **Severity tags to focus your review** * Critical and Major issues float to the top with our new severity tags. Refactors are gated. Nitpicks are opt-in. You will spend less time scanning comments and more time fixing. 4. **A faster feedback loop** * Codex’s leaner reasoning plus pipeline improvements bring time to first helpful comment down. You will feel it. ## **Quantitative appendix (for the curious)** We know you love data! Here’s some other stats we found interesting: * **Per comment precision (SNR) uplift:** Codex 46.3% vs. GPT‑5 34.2% — about +35% relative. * **Comment volume delta:** Codex 54 vs. GPT‑5 79 — 32% fewer comments, with EP passes essentially unchanged (16 vs. 15). * **Style:** Codex includes diffs in 94% of comments and uses hedging far less than Claude and GPT‑5 on this set. * **Acceptance (real world):** During GPT‑5 rollout, acceptance dropped significantly. With Codex plus changes, it rose by about 20–25% relative and returned to prior levels while still delivering more accepted comments than pre GPT‑5. ## **Where Codex still needs work (and what we are doing)** These improvements are great but that doesn’t mean that there aren’t still issues with Codex. Here are some that we are actively working on: * **Coverage gaps.** When a model leaves no comment on a PR, that is a hard fail for that EP. We are widening Codex’s search heuristics so it is less likely to miss entire classes of issues. * **Refactor over eagerness (tuned, not solved).** The “essential only” gate curbs refactor noise, but we will keep tightening the threshold, especially on large diffs where a high number of comments would be overwhelming. * **User driven prioritization.** We cannot change GitHub’s in-line ordering, but we annotate every comment with severity so you can triage from the top down without hunting. ## **Codex GPT-5: All of the great bug catching ability, fewer downsides** Our north star is simple: **catch the bugs that matter, quickly, without making you sift through noise**. Codex helps us do that. It keeps the bite of GPT‑5’s reasoning while restoring SNR and shaving latency down significantly. We will keep measuring, improving, and shipping a better product every release.

We raised $60 million last week… so we made a funny video

Aravind Putrevu — Tue, 23 Sep 2025 00:00:00 GMT

Last week, we announced CodeRabbit’s **$60 million Series B**. To celebrate, we did what any responsible, developer-focused software company would do: we made a funny video. For the full announcement, read about [building quality gates for AI-powered coding](https://www.coderabbit.ai/blog/coderabbit-series-b-60-million-quality-gates-for-code-reviews). Not with *all* the money, to be clear. But we did decide to celebrate with something fun, absurd, and painfully relatable for any dev team trying to keep up with the flood of AI-generated PRs. ## **Introducing… “When AI coding agents backfire: A short film”** %[https://youtu.be/glfB3KLQR7E?feature=shared] It’s a short mockumentary-meets-sitcom about what really happens when “AI velocity” turns into a PR review backlog. * One reviewer. * Dozens of notifications. * 84 open PRs. * And one overly eager coworker named Brad who just wants feedback. ## **The cast** ![We raised $60 million last week… so we made a funny video](/content/assets/we-raised-60-million-last-week-so-we-made-a-funny-video-inline-image.png) To bring it to life, we pulled in beloved developer educator (and influencer) [**Aaron Francis**](https://x.com/@aarondfrancis) to star as our beleaguered reviewer. He’s the guy who just wanted to ship features faster and now can’t go to the kitchen (or even leave his house at 8 a.m.) without Brad asking about his PR. And speaking of Brad: the inimitable [**Austin von Johnson**](https://www.instagram.com/4ustinvon/?hl=en) plays him to perfection. Brad’s a developer who can crank out AI-generated PRs at lightning speed but cannot, under any circumstances, *wait patiently* for a review. His lurking, his post-its, his hoodie PR ambushes… let’s just say he was perfectly committed to the bit. ## **The very real problem behind the joke** ![We raised $60 million last week… so we made a funny video](/content/assets/we-raised-60-million-last-week-so-we-made-a-funny-video-inline-image-2.png) The short film is funny, but the problem it highlights is real: * **AI coding tools crank out code faster than teams can review it** * **Review backlogs balloon** while productivity drops * **Senior engineers get buried** in endless PRs * Review quality is uneven, risk goes up, and you have to deal with more issues * And suddenly, the promise of velocity feels more like a nightmare. ## **Here’s what we’re doing about it** CodeRabbit exists to **clear the backlog**, not add to it. Our AI code reviews pull in dozens of points of [context](https://www.coderabbit.ai/guides/code-context) (requirements, tests, CI, past diffs, ownership) to catch bugs you’d miss, reduce reviewer fatigue, and move PRs through faster—without turning teammates into… Brad. Ship faster, review smarter, and keep your sanity. Also, avoid creating a… Brad. Watch **“When AI coding agents backfire: a short film”** in the video above. And if you’ve ever been chased around the office about a PR, please, send it to your team’s Brad.

CodeRabbit commits $1 million to open source

Aravind Putrevu — Thu, 18 Sep 2025 00:00:00 GMT

Open source is the foundation of modern software development. From package managers and developer tools to frameworks and infrastructure, open source projects power nearly every piece of software we use today – including CodeRabbit itself. These projects are built and maintained by communities of developers who dedicate countless hours to keeping them alive, secure, and evolving. Today, we’re proud to announce a **$1 million USD commitment to open source software sponsorships**. This commitment comes on the heels of our **$60 million Series B funding round** and it reflects both our gratitude for what open source makes possible and our belief in the importance of investing in its future. ## **Why open source needs support now more than ever** Generative AI is transforming software development, but it’s also putting new pressures on open source maintainers. Alongside the surge of high-quality contributions, there has been a sharp rise in **AI-generated PR spam**: repetitive, low-quality, and sometimes insecure code submissions that overwhelm project maintainers. We’ve heard firsthand from maintainers about how draining this flood of noise can be. At CodeRabbit, we’ve built tools that **filter out spam, elevate code quality, and reduce maintainer workload** by blending AI-driven code review with human oversight. We’ve made our AI code review tool free for use on all open source projects (more about that here). But tools alone aren’t enough—sustainable open source requires financial support, recognition, and stronger bridges between communities. ## **From $200K to $1M: Deepening our commitment** Earlier this year, we announced a [**$200,000 pledge to open source**](https://2025.allthingsopen.org/pledging-support-for-open-source), supporting projects like: * **pnpm**: A disk-space–efficient package manager * **Biome (biomejs)**: A next-generation linter and formatter for JavaScript and TypeScript * **AST Grep (Herrington Darkholme)**: Structural code search for smarter code analysis * **iTerm2 (George Nachman)**: A terminal emulator that redefined developer workflow * **Markdown Lint (David Anson)**: Ensuring docs stay clear and consistent That pledge was only the beginning. With our new Series B funding, we’re scaling our support to **$1 million**, ensuring that more projects and maintainers across the ecosystem receive the recognition and resources they deserve. [**Apply for sponsorship here.**](https://coderabbit.link/oss-progam-submission-form) ## **CodeRabbit & OSS: Building bridges across the OSS ecosystem** Sponsorship is only part of the story. Many of the challenges open source faces—sustainability, security, and developer burnout—aren’t isolated to a single project. They stretch across communities and ecosystems. That’s why CodeRabbit is also working to **connect maintainers, foster collaboration, and share solutions across projects**. Whether through joint sponsorships, shared initiatives, or community-driven tooling conversations, we aim to strengthen the ecosystem as a whole rather than supporting it in silos. If you’re a maintainer or contributor who wants to join these conversations, we’d love to hear from you. Join our Discord to connect with the CodeRabbit team and other open source leaders. ## **Free CodeRabbit access for open source projects** Finally, a reminder: **CodeRabbit is free for open source**. Every maintainer, contributor, and community can use our platform to cut through PR noise, automate code quality checks, and free up more time for meaningful contributions. ***Learn more and*** [***apply for funding here.***](https://coderabbit.link/oss-progam-submission-form)

CodeRabbit’s MCP integration = Code reviews that see the whole picture

Edgar Cerecerez — Wed, 17 Sep 2025 00:00:00 GMT

Every dev team knows the pain of code reviews if performed in isolation. An AI tool (or even a teammate) can comment on syntax, style, and patterns, but without business requirements, deployment dependencies, or organizational knowledge, it’s just guessing at half the story. CodeRabbit currently has a number of native integrations including Linear, Jira, and Circle CI. We have seen the value that context from those tools provide to code reviews. That’s why we’re excited to announce the GA of **CodeRabbit’s integration with MCP servers**. This will allow you to bring in even more context into your reviews. With this launch, we become the first AI code review platform that orchestrates context from across your entire development ecosystem from business requirements in Confluence to system dependencies in your CI/CD pipeline to data from any internal MCP servers. All to provide code reviews that actually understand what your code is trying to accomplish. **Start your 14-day trial →** *Get context-aware reviews that reference your actual team standards in ~10 minutes.* ## **Why MCP for AI code reviews?** Development teams operate across dozens of tools: * Requirements live in Linear * Design specifications exist in Figma * Architectural decisions get documented in Confluence * Security standards evolve in internal wikis after each audit AI code reviewers start with basic context: your codebase, some coding guidelines, maybe a few integrations. They analyze syntax, check patterns, and suggest improvements. But they miss the context that determines whether code actually works for your team. As a MCP client, CodeRabbit acts as a compiler for organizational context. It takes high-level inputs - your wikis, tickets, deployment patterns - and compiles them down into precise, actionable code review insights. Instead of bloated integrations or brittle hacks, MCP lets clients like CodeRabbit pull in just the right data from your MCP servers from places like your Linear tickets, Confluence docs, Datadog metrics, or Slack discussions. ## What it looks like in practice… CodeRabbit searches connected MCP servers before starting a review. For example, database schema changes might get checked against data architecture documents. API endpoint implementations might get verified against service design patterns documented in internal wikis. **Example: CodeRabbit verifies code consistence** ![CodeRabbit’s MCP integration = Code reviews that see the whole picture](/content/assets/coderabbits-mcp-server-integration-code-reviews-that-see-the-whole-picture-inline-image.png) ## Bring in the context matters to you… from any tool ![CodeRabbit’s MCP integration = Code reviews that see the whole picture](/content/assets/coderabbits-mcp-server-integration-code-reviews-that-see-the-whole-picture-asset.png) Traditional code review tools require specific integrations. CodeRabbit's MCP integration works with any system with an MCP server. Your proprietary internal tools, boutique SaaS platforms, custom documentation systems. If there's an MCP server, CodeRabbit can connect. With CodeRabbit as an MCP client, you’re reviews gain depth from bringing in three different types of context. ### **Technical context.** * Think dependencies, performance data, static analysis, and test coverage. * **Native integrations:** GitHub Actions, GitLab CI, Bitbucket Pipelines * **MCP Servers:** Datadog, New Relic, SonarQube, Snyk, Grafana * Example Review Comment: ![CodeRabbit’s MCP integration = Code reviews that see the whole picture](/content/assets/coderabbits-mcp-server-integration-code-reviews-that-see-the-whole-picture-asset-2.png) ### **Business context.** * This includes things like requirements, user stories, and acceptance criteria. * **Native integrations:** Linear, Jira, GitHub Issues, GitLab Issues * **MCP Servers:** Confluence, Notion * Example Review Comment: ![CodeRabbit’s MCP integration = Code reviews that see the whole picture](/content/assets/coderabbits-mcp-server-integration-code-reviews-that-see-the-whole-picture-asset-3.png) ### **Organizational context.** * We also pull in things like prior decisions, conventions, meeting notes, and institutional knowledge. * **Native integrations:** PR history, Team conventions * **MCP Servers:** Slack, Microsoft Teams, Stack Overflow for Teams, PagerDuty * Example Review Comment: ![CodeRabbit’s MCP integration = Code reviews that see the whole picture](/content/assets/coderabbits-mcp-server-integration-code-reviews-that-see-the-whole-picture-asset-4.png) ## **Getting started with MCP integration** Setting up CodeRabbit's MCP client requires minimal configuration. Most development teams can connect their first MCP server in under 10 minutes. **Popular development tools with MCP server support**: * **Linear** (native MCP support, 5 minutes) * **Notion** (MCP server available, 10 minutes) * **Confluence** (community MCP server, 15 minutes) * **Figma** (MCP plugin available, 10 minutes) Define which code changes should search which development systems. Database changes check architecture documentation. Authentication changes check security documentation. Adding an MCP server is easy: 1. In the CodeRabbit dashboard, head over to integrations > and toggle to the MCP Servers tab if needed ![CodeRabbit’s MCP integration = Code reviews that see the whole picture](/content/assets/coderabbits-mcp-server-integration-code-reviews-that-see-the-whole-picture-asset-5.png) 2. You can click on one of the pre-configured MCP server options or the New MCP Server button to add other MCP servers. ![CodeRabbit’s MCP integration = Code reviews that see the whole picture](/content/assets/coderabbits-mcp-server-integration-code-reviews-that-see-the-whole-picture-asset-6.png) 3. For MCP servers not on the list, enter the relevant credentials. 4. Note the usage guidance which serves as context for how the MCP information should be used. ![CodeRabbit’s MCP integration = Code reviews that see the whole picture](/content/assets/coderabbits-mcp-server-integration-code-reviews-that-see-the-whole-picture-asset-7.png) 5. Once connected. You can see the available calls and hover over them to see more details. ![CodeRabbit’s MCP integration = Code reviews that see the whole picture](/content/assets/coderabbits-mcp-server-integration-code-reviews-that-see-the-whole-picture-asset-8.png) 6. You can also click on each call to enable/disable access. ![CodeRabbit’s MCP integration = Code reviews that see the whole picture](/content/assets/coderabbits-mcp-server-integration-code-reviews-that-see-the-whole-picture-asset-9.png) > Note: All MCP Server queries are ephemeral. CodeRabbit processes them in real-time with zero data retention. ## **A review platform that brings in all your context** CodeRabbit works out of the box with 50+ integrations. With MCP, you can extend it to your custom servers and internal tools. Start with the systems you already use — Linear, Confluence, Datadog, Slack — and add more as you go. ### **Next steps:** 1. [**Start a 14-day trial**](https://app.coderabbit.ai/login?free-trial=) 2. [**View MCP server directory**](https://app.coderabbit.ai/integrations) 3. [**See the MCP docs**](https://docs.coderabbit.ai/context-enrichment/mcp-server-integrations)

CodeRabbit CLI - free AI code reviews in your CLI

Edgar Cerecerez — Tue, 16 Sep 2025 00:00:00 GMT

CodeRabbit started with AI-powered code reviews in pull requests. In May, we brought that same intelligence to [VS Code, Cursor, and Windsurf](https://www.coderabbit.ai/blog/ai-code-reviews-vscode-cursor-windsurf). Now, we're extending the AI code reviews developers love into the command line with CodeRabbit CLI. In case you’re wondering, that makes us the most comprehensive AI code review tool available. We work everywhere you work. *CodeRabbit CLI* helps devs perform self-reviews of code directly in their terminal. By providing automated, intelligent code analysis capabilities, it empowers developers to catch issues early, maintain consistent code standards, and make coding autonomous through seamless integration with AI coding agents in the CLI. ## **Vibe checking your code – now in CLI** %[https://youtu.be/IqBKf4u5MtA] CodeRabbit CLI delivers the same comprehensive analysis that makes our PR and IDE reviews effective at catching bugs early. CodeRabbit CLI is free to use with rate limits but with a Pro plan you can enjoy much higher limits and additional features, including: * **Context-aware analysis**: Leverages your Git integration to synthesize insights from 40+ sources including static analysis tools, security scanners, and our codegraph relationship feature for the most comprehensive reviews. * **Pre-commit reviews**: Analyze changes before they leave your machine for multi-layered reviews. * **One-click fixes**: Apply simple fixes instantly or send complex issues to AI agents with full context hand-off. * **Coding guidelines**: Auto-detects agent.md, claude.md, Cursor rules, and other coding agent configuration files. ## **CodeRabbit CLI: Works everywhere, with everything** Terminal-native means CodeRabbit CLI works with: * **Any Terminal App/IDE:** iTerm2, Ghostty, Neovim, Lazyvim * **Any AI Coding CLI agent**: Claude Code, Codex, Cursor, Gemini, OpenCode and more ## **How to use CodeRabbit CLI with AI Coding Agent CLI** The CodeRabbit CLI opens up new integration possibilities with [AI coding agents](https://www.coderabbit.ai/guides/types-of-ai-coding-agents). Here's how it works with Claude Code: 1. While working on a coding task, you can prompt Claude Code to use CodeRabbit and to fix any issues it finds. This is particularly useful if it’s coding from a PRD, or a tasklist. ```plaintext Please implement phase 7.3 of the planning doc and then run coderabbit --prompt-only, let it run as long as it needs (run it in the background) and fix any issues. ``` ![CodeRabbit CLI - Free AI code reviews in your CLI](/content/assets/coderabbit-cli-free-ai-code-reviews-in-your-cli-inline-image.png) 2\. Claude Code will carry on the coding task and run `coderabbit --prompt-only` in the background. It may setup a timer interval to check on CodeRabbit. Alternatively, you can also prompt Claude to check if CodeRabbit is complete. ![CodeRabbit CLI - Free AI code reviews in your CLI](/content/assets/coderabbit-cli-free-ai-code-reviews-in-your-cli-asset.png) ![CodeRabbit running in the background within Claude Code](/content/assets/coderabbit-cli-free-ai-code-reviews-in-your-cli-asset-2.png) 3\. Claude Code will then read the output of CodeRabbit which, by using the `--prompt-only` flag, provides the output as plain text with prompts for AI agents to read. Claude will then create a tasklist addressing each of the issues surfaced by CodeRabbit. ![CodeRabbit CLI - Free AI code reviews in your CLI](/content/assets/coderabbit-cli-free-ai-code-reviews-in-your-cli-asset-3.png) ![CodeRabbit CLI - Free AI code reviews in your CLI](/content/assets/coderabbit-cli-free-ai-code-reviews-in-your-cli-asset-4.png) For Claude Code integration and automated workflows, check the [CLI documentation](https://docs.coderabbit.ai/cli) [for setup.](https://docs.coderabbit.ai/cli) The CLI has two modes: interactive and plain response , making it easy to integrate into automated workflows or pass results to other tools. ## **Getting started** CodeRabbit CLI is available now. Install and try your first review: ```powershell #install CodeRabbit curl -fsSL https://cli.coderabbit.ai/install.sh | sh #Run a review in interactive mode coderabbit ```

Raising our $60M Series B: building the quality gates for AI-powered coding

Harjot Gill — Tue, 16 Sep 2025 00:00:00 GMT

When we started CodeRabbit, the idea was pretty simple: since all developers hate code reviews, why not make them faster and easier? After all, no one enjoys leaving the same comment about variable naming practices or style conventions for the tenth time in a week. That’s where we believed AI could help – it could automate best-practice checks and policy enforcement so that devs didn’t have to do it themselves. But more importantly, it could act as a safety net, catching issues and bugs before they made it into production. With that belief, we set out to [create something new: AI code reviews.](https://www.coderabbit.ai/blog/coderabbit-announces-16m-series-a-funding-led-by-crv) Over time, AI coding tools started to gain broader adoption. Tools like Copilot, Claude Code, and Cursor began spitting out more code than teams could easily review with many developers increasing the number of PRs they shipped by 2x to 3x. This added to the existing code review backlogs many teams had. We quickly realized that the ‘efficiency’ gains being marketed to engineering teams would swiftly turn into code review bottlenecks. And that’s also when we first realized how critical AI code reviews would be to development teams. They would function as a trust and governance layer in agentic software development ensuring quality and security while saving devs time. For the deeper framework behind that idea, see [AI-generated code quality gates](https://www.coderabbit.ai/guides/ai-generated-code-quality-gate). And, as an added bonus, greatly reducing passive-aggressive review comments in the workplace! ## **AI code reviews became essential in 2025** %[https://youtu.be/UHCTKZYOOYU] Over the last two years, we’ve built the most comprehensive and context-rich platform for code reviews, been installed on 2 million repos, reviewed 13 million pull requests, become the most installed AI App on both GitHub and GitLab, and improved the morale of countless dev teams. In 2025, we watched AI code reviews become essential for all teams dealing with the challenges that come with the broad adoption of AI coding agents. But that shift fueled a year of unprecedented growth, culminating in the **$60 million Series B round** that we announced today. This investment was led by **Scale Venture Partners** with participation by **NVentures (NVIDIA’s venture capital arm)** and support from our long-time investors **CRV, Harmony Partners, Flex Capital, Engineering Capital, and Pelion Venture Partners**. With this new funding, our total capital raised is now $88 million. ## **Why so many teams are adopting AI code reviews** When every developer on your team is generating code faster, your review queue grows exponentially. Senior engineers who used to review 5 to 10 PRs a day are now facing 20 to 30. The math doesn't work. Teams are caught between two bad options: either slow down deployment cycles waiting for thorough reviews, or rush reviews and let quality slip. That same pressure shows up when teams pair AI code generators with AI code reviewers. This is why AI code review adoption is accelerating. AI reviewers augment the human reviewers, freeing them to focus on architecture decisions, business logic, and the nuanced feedback that requires context AI can't fully grasp yet. The past year has been a whirlwind. We’ve 10x revenue and doubled our team thanks to: ![Raising our $60M Series B: Building the quality gates for AI-powered coding](/content/assets/coderabbit-series-b-60-million-quality-gates-for-code-reviews-asset.png) Behind each of those customers are real teams who tell us the same thing: reviews are faster with CodeRabbit, bugs are caught earlier, and release cycles are finally speeding up again. Groupon told us they went from 86 hours from review-to-production down to just 39 minutes. Another shared that they cut down the time they spend on code reviews by 70%. ## How CodeRabbit works CodeRabbit works because it fights AI fire with AI fire. Our platform brings in dozens of points of context to deliver the most context aware reviews to: * Catch correctness and security issues before they hit production. ![Raising our $60M Series B: Building the quality gates for AI-powered coding](/content/assets/coderabbit-series-b-60-million-quality-gates-for-code-reviews-asset-2.png) * Enforce organizational best practices and custom policies. ![Raising our $60M Series B: Building the quality gates for AI-powered coding](/content/assets/coderabbit-series-b-60-million-quality-gates-for-code-reviews-asset-3.png) * Support the full merge cycle with unit testing and docstrings generation. ![Raising our $60M Series B: Building the quality gates for AI-powered coding](/content/assets/coderabbit-series-b-60-million-quality-gates-for-code-reviews-asset-4.png) ## How we’re celebrating: By announcing CodeRabbit CLI %[https://youtu.be/IqBKf4u5MtA] Today, we're celebrating our Series B by announcing CodeRabbit CLI, AI code reviews that live in your terminal and orchestrate seamlessly with Claude Code, Codex CLI, Cursor CLI, Gemini, and other AI coding agents. As developers increasingly write code through CLI Coding agents, we've identified a critical gap: code is being generated at unprecedented speeds, but quality validation happens too late, often only at the PR stage. CodeRabbit CLI changes this by bringing intelligent review directly into the CLI workflow, creating a real-time feedback loop between code generation and validation. Now, whether you're prompting Claude Code to refactor a module or using Cursor CLI to implement a feature, CodeRabbit instantly reviews the output, catches hallucinations, flags security issues, and even hands contextualized fixes back to your AI agent. CodeRabbit CLI is the missing orchestration layer that makes AI-generated code production-ready, turning the promise of autonomous development into reality. ![Raising our $60M Series B: Building the quality gates for AI-powered coding](/content/assets/coderabbit-cli-free-ai-code-reviews-in-your-cli-asset.png) ## **What this funding means for us (and for you)** Our Series B round will help us keep pace with the scale of the problem we set out to solve. Here’s where we’re putting that investment: * **Accelerating product development:** Our roadmap is packed. From deeper context integrations to smarter pre-merge checks and automated testing, we’re focused on making reviews faster, more accurate, and more useful for every team. * **Supporting open source:** Today, more than 100,000 OSS projects already use CodeRabbit. With this funding, we’re doubling down on contributions and support to strengthen the community that made modern development possible. More on that later in the week! * **Hiring the best talent:** We’ve already doubled headcount this year and we’re hiring globally across engineering, product, sales, marketing, and customer success. This funding gives us the space to keep building what we believe is the most important missing piece of AI-powered development: scalable, context-aware reviews. ## **Thank you for all your support** When we started this company, we knew we were chasing a problem every engineer experiences: reviews are a pain and they don’t scale easily. The fact that CodeRabbit is now helping thousands of teams tackle that problem is both humbling and energizing. To our customers, community, and investors: thank you for believing in us and building alongside us. And if this work excites you, consider joining us. Come help us build the future of code reviews. [***Try CodeRabbit for free***](https://coderabbit.link/KWGzOUS) ***yourself and learn more about our*** [***open roles.***](https://coderabbit.link/OG1OZk3)

How CodeRabbit delivers accurate AI code reviews on massive codebases

Sahana Vijaya Prasad — Fri, 05 Sep 2025 00:00:00 GMT

Massive codebases are a special kind of beast. They sprawl across hundreds of files, evolve over years of commits, and occasionally feel like they’re held together by equal parts duct tape and institutional memory. Reviewing changes in that environment isn’t just hard – it feels like an archaeological dig. Did this line move here last week for a reason? Is there another file quietly depending on it? That’s exactly where CodeRabbit shines. It was built for scale, so instead of drowning you in disconnected file-by-file comments, it reviews with the whole history and architecture of your massive codebase in mind. The larger and older your repository, the more useful CodeRabbit becomes because it can see the patterns, dependencies, and rules that humans usually forget about halfway through a pull request when trying to keep all the dependencies in that legacy code in their head. For large PRs, [Change Stack](https://www.coderabbit.ai/blog/introducing-change-stack-the-first-ai-native-code-review-interface) turns that context into a guided review path. ## Large codebase? AI code reviews need more context! ![How CodeRabbit delivers accurate AI code reviews on massive codebases](/content/assets/how-coderabbit-delivers-accurate-ai-code-reviews-on-massive-codebases-inline-image.png) CodeRabbit is [known for performing great on large repos](https://x.com/qwertyu_alex/status/1956848505445654595). Our tool doesn’t just skim your pull requests; it goes full archivist. Before leaving a single comment, it gathers the surrounding code from your large codebase and pulls in dozens of points of context from your code. AI agents then trace how those pieces have moved through history, apply your team’s coding standards, and even double-check their own reasoning with scripts and tools. For a closer look at that evidence model, see [Code context for trustworthy AI code review](https://www.coderabbit.ai/guides/code-context). The effect is reviews that feel unusually…informed about your legacy codebase. It catches cross-file issues before they turn into production mysteries, enforces consistency without nitpicking, and scales comfortably across sprawling repos with long, complicated pasts. The power you gain through this is clearer, earlier feedback on real risks, fewer “wait, what else did that touch?” surprises, and reviews that actually reflect how your whole massive codebase fits together. ## **The problem with diff-only reviews (or what goes wrong without context)** Code diffs are necessary, but they’re not sufficient. In a massive codebase, a 10-line change can quietly alter a shared helper used by multiple services, shift a public API contract, or undermine a security assumption that lives outside the files in the diff. That is the same review problem behind [large AI-generated migrations](https://www.coderabbit.ai/blog/ai-can-migrate-your-entire-codebase-reviewing-it-is-another-story). AI Bot reviewers who only see the diff are flying without instruments within a large codebase. AI that can’t see where the changed code is referenced, what else tends to change with it, or whether the change actually matches the ticket’s intent, might work for a smaller codebase but not for yours. Without the right context, you get ping-pong cycles (“Can you also update…?”), late surprises at merge time, and a steady drip of small regressions that add up. The review looks fine on paper, while production tells a different story. ## **Building the right context on your legacy codebase (and how that helps your PRs)** Think of CodeRabbit as assembling a case file before giving an opinion. Here’s what goes into that case file and how each piece shows up in your reviews. 1. ### **A map of your code (Codegraph)** ![How CodeRabbit delivers accurate AI code reviews on massive codebases](/content/assets/how-coderabbit-delivers-accurate-ai-code-reviews-on-massive-codebases-asset.png) CodeRabbit builds a lightweight map of definitions and references and scans commit history for files that frequently change together throughout your massive codebase. This creates a map of file dependencies that CodeRabbit uses to check if any changes in your PR will break other dependencies in your codebase. **Why this helps:** The review can reason across files, not just lines. **Seeing it in action:** CodeRabbit posts a summary listing bugs **outside the diff range** that CodeRabbit located by traversing related files with Codegraph. **Here’s an example of the files that CodeGraph brings in from across a repository when completing a PR review.** ![How CodeRabbit delivers accurate AI code reviews on massive codebases](/content/assets/how-coderabbit-delivers-accurate-ai-code-reviews-on-massive-codebases-asset-2.png) 2. ### **Code Index (semantic & similarity retrieval)** ![How CodeRabbit delivers accurate AI code reviews on massive codebases](/content/assets/how-coderabbit-delivers-accurate-ai-code-reviews-on-massive-codebases-asset-3.png) CodeRabbit maintains a semantic index (embeddings) of functions, classes/modules, tests, and prior PRs/changes. During review, it searches by purpose, not just keywords to surface parallel implementations to align with, pull relevant tests to reuse or extend, and recall how similar issues were fixed before. **Why this helps:** Suggestions are grounded in how your legacy codebase already solves similar problems, reducing rework, improving consistency, and speeding up test coverage. **Seeing it in action:** Using similarity retrieval, CodeRabbit surfaces a different test with the same callback pattern and proposes the same fix. 3. ### **Your team rules, not generic advice** ![How CodeRabbit delivers accurate AI code reviews on massive codebases](/content/assets/how-coderabbit-delivers-accurate-ai-code-reviews-on-massive-codebases-asset-4.png) CodeRabbit reviews are primed with your standards (naming, error handling, API boundaries, security requirements, performance expectations, testing norms) that you can share with us via coding guidelines and review instructions. **Why this helps:** Feedback reflects *your* standards and context, not a one-size-fits-all checklist. **Seeing it in action:** CodeRabbit flags a missing Prisma migration after a schema edit. A developer replies that migrations are auto-generated during deploy, a repo-specific rule. CodeRabbit stores that as a **Learning** to avoid future false positives. 4. ### **Signals from tools** ![How CodeRabbit delivers accurate AI code reviews on massive codebases](/content/assets/how-coderabbit-delivers-accurate-ai-code-reviews-on-massive-codebases-asset-5.png) Alongside AI reasoning, CodeRabbit runs linters and security analyzers and folds their findings into our easy-to-read and understand reviews. **Why this helps:** You get grounded, actionable suggestions backed by both AI *and* recognizable tools. **Seeing it in action:** CodeRabbit will do things like point to the exact ESLint rule and line numbers, rewrites the callback as a typed declaration, and guards the call with optional chaining. 5. ### **Evidence, not vibes (verification scripts)** ![How CodeRabbit delivers accurate AI code reviews on massive codebases](/content/assets/how-coderabbit-delivers-accurate-ai-code-reviews-on-massive-codebases-asset-6.png) When something needs checking, CodeRabbit generates shell/Python checks (think grep, ast-grep) to confirm an assumption or extract proof from the codebase before we post the comment. **Why this helps:** Comments come with ***receipts***. That translates into less noise and more comments that actually improve your code. **Seeing it in action:** The comment pinpoints the file and loop, explains the failure mode, and proposes the exact change produced by the verification agent after analyzing the parsing path. This is **context engineering in practice**: gathering, filtering, and organizing the *right* information before asking the model to judge. It’s been core to CodeRabbit since day one. The payoff is simple: higher signal, lower noise, and reviews that feel like they understand your system. ## **Scaling to enterprise-size repos** CodeRabbit has an advantage on massive codebases and legacy codebases because we designed our pipeline with scale in mind. When a PR arrives, CodeRabbit spins up an isolated, secure, short-lived environment to do the work. It pulls only what it needs, constructs the context, runs the checks, and tears everything down after. During busy hours, many of these workers run in parallel so review speed holds steady. You stay in control of scope by using path filters to keep bulky or generated assets out of the way, and choosing whether to enable caching or indexing to accelerate repeat reviews. In short: selective scope keeps context focused, isolation keeps it safe, and elastic execution keeps it fast. This approach scales with your codebase and your release calendar. ## **CodeRabbit: Large codebase AI code reviews done right** CodeRabbit’s advantage on massive codebases isn’t a single trick. It comes from how we approach context engineering end-to-end: map what the change touches, tie it to intent, apply your rules, verify with tools, then comment with evidence. We’ve operated this way from the start, well before “context engineering” became a buzzword, because it’s the only reliable path to accurate, low-noise reviews at scale. ***Ready to see a deep-context review on your large codebase? →*** [***Start a 14-day trial***](https://coderabbit.link/sY5vXpT)

Our response to the January 2025 Kudelski Security vulnerability disclosure: action and continuous improvement

Harjot Gill — Tue, 19 Aug 2025 00:00:00 GMT

## No customer data was accessed and the vulnerability was quickly remediated within hours of disclosure As the CEO, I want to address recent reports of a security vulnerability discovered in January 2025 by Kudelski Security researchers and share our immediate response, the steps we've taken since, and our ongoing commitment to security. ## **What happened** ![Our response to the January 2025 Kudelski Security vulnerability](/content/assets/our-response-to-the-january-2025-kudelski-security-vulnerability-disclosure-action-and-continuous-improvement-inline-image.png) On **January 24, 2025,** security researchers from Kudelski Security disclosed a vulnerability to us through our Vulnerability Disclosure Program (VDP). The researchers identified that RuboCop, one of our tools, was running outside our secure sandbox environment—a configuration that deviated from our standard security protocols. We immediately initiated an investigation and were able to remediate this issue within hours through our rapid incident response protocol. We confirmed the issue disclosed by Kudelski Security and that there was no evidence of any other unauthorized access, identified the root cause, implemented a fix, and, as described below, enhanced our comprehensive security protocols to prevent similar incidents. To be clear: **We use secure sandboxes as standard practice across our infrastructure.** This was an oversight on our part and we take full responsibility for it. ## **Our immediate response** Upon receiving the disclosure, our security team activated our incident response protocol: * **Within 1 hour**: We confirmed the vulnerability and began immediate remediation by first disabling RuboCop until we could fix the vulnerability. * **Within 3 hours**: We completed a full rotation of all relevant credentials and secrets. * **Within 12 hours**: We deployed a comprehensive fix to production, relocating RuboCop into our secure sandbox environment. * **Additionally, we**: * Conducted a thorough audit of all systems to ensure no other services were running outside our sandbox infrastructure. * Automated sandbox enforcement. * Introduced enhanced deployment gates. * Audited and updated our mandatory security training for all engineers. **We promptly investigated to identify any potential unauthorized access. The investigation identified no evidence that any customer data was accessed or that any malicious activity occurred.** ## **Why this matters to us** Security isn't just a checkbox for us; it's fundamental to our mission. While our services run within secure sandboxes as designed, in this case, the investigation determined that RuboCop had been deployed outside this security boundary. This deviation from our standards, while contained quickly and without customer impact, is unacceptable to us. We took action immediately to ensure it wouldn’t happen again. **What we're doing differently** 1. **Comprehensive sandbox audit**: We immediately completed a full review of ALL services to ensure 100% compliance with our sandbox requirements. RuboCop was the only service found outside our sandbox environment, and this has been rectified. 2. **Automated sandbox enforcement**: We immediately implemented automated checks that have since prevented any service from deploying outside our security boundaries. 3. **Enhanced deployment gates**: Every deployment now requires supplemental explicit sandbox verification before reaching production. 4. **Ongoing training verification:** We added recurring checks to confirm that all engineers complete the updated mandatory security training. ## **Our VDP program: Security through collaboration** This vulnerability disclosure exemplifies why we've invested heavily in building a Vulnerability Disclosure Program. It features: * **Active researcher engagement**: We maintain ongoing relationships with multiple security researchers worldwide. * **Competitive rewards**: Top-tier bounties that recognize the value of security research. * **Fast response times**: Average first response under 24 hours, resolution within 7 days. * **Clear communication**: Dedicated security team providing regular updates throughout the disclosure process. ## **The value of responsible disclosure** Kudelski Security's professional approach allowed us to address this vulnerability before it could be exploited maliciously. This is exactly how the security ecosystem should work—researchers and companies collaborating to improve security for everyone. We're grateful for their professionalism and encourage all security researchers to engage with us through our VDP program at [https://vdp.coderabbit.ai/](https://vdp.coderabbit.ai/). Whether you're an independent researcher or part of an established firm, we value your contributions to our security. ## **Our commitment** To our users, we will continue to: * Maintain secure sandboxes as our default security boundary for all services * Invest heavily in security infrastructure and tooling * Run one of the industry's most comprehensive VDP programs * Actively engage and reward security researchers * Learn from every vulnerability disclosure and incident, no matter how small * Hold ourselves to the highest security standards * Maintain compliance with industry security standards like SOC 2 Type 2 We're grateful to Kudelski Security for their research and committed to our users who trust us with their data. We welcome any questions or concerns at [security@coderabbit.ai](mailto:security@coderabbit.ai) or through our VDP portal at [https://vdp.coderabbit.ai/](https://vdp.coderabbit.ai/).

Benchmarking GPT-5: Why it’s a generational leap in reasoning

Christopher Cassion — Thu, 07 Aug 2025 00:00:00 GMT

The wait is over! As the leading AI code review tool, CodeRabbit was given early access to OpenAI’s GPT-5 model to evaluate the LLM’s ability to understand, reason through, and find errors in complex codebases. As part of our GPT-5 testing, we've conducted extensive evals to uncover its technical nuances, capabilities, and use cases with a focus on the model’s ability to understand and reason through potential issues and bugs in codebases. Below, you’ll find a breakdown of our structured evaluation approach, detailed findings relative to other popular models, and how we’re planning to incorporate GPT-5 into your AI code reviews to make them even better. ## TL;DR: The results ![Benchmarking GPT-5: Why it’s a generational leap in reasoning](/content/assets/benchmarking-gpt-5-why-its-a-generational-leap-in-reasoning-inline-image.png) * GPT-5 outperformed Opus-4, Sonnet-4, and OpenAI’s O3 across a battery of 300 varying difficulty, error-diverse pull requests. * GPT-5 scored highest on our comprehensive test and found 254 out of 300 bugs or 85% where other models found between 200 and 207 – 16% to 22% less. * On our 25 hardest PRs from our evaluation dataset, GPT-5 achieved the highest ever overall pass rate (77.3%), representing a **190% improvement over Sonnet-4**, **132% over Opus-4**, and **76% over O3**. ## How we evaluated GPT-5 ![Benchmarking GPT-5: Why it’s a generational leap in reasoning](/content/assets/benchmarking-gpt-5-why-its-a-generational-leap-in-reasoning-inline-image-2.png) We ran the same tests we run on all our models. These evals integrate GPT-5 into our context-rich, non-linear code review pipeline to see how it would perform in a typical code review. CodeRabbit's evaluation process includes: * **LLM-based judging:** We perform dual-layered LLM-based judgment that looks at both qualitative and quantitative data such as the quality of a review and a pass/fail of the model’s accuracy. * **Human-based judging:** We then perform qualitative checks by humans to verify the quality of review comments and depth of the model’s reasoning. * **LLM-based metrics collection:** We collect metrics that we believe are indicative of a high quality code review and weigh them by their importance. These metrics include: * Actionable comment counts * Readability scores (Flesch Reading Ease score) * Average word count * Sentence count * False positives (hallucinations) **Note:** Our evaluations were conducted on various ‘snapshots’ of GPT-5 that OpenAI shared with us leading up to the release of GPT-5. While our results changed somewhat with different snapshots, their relative consistency allowed us to make the observations below. The released model might be slightly different. ## GPT-5 capabilities: Evaluation results and analysis Our evaluation of GPT-5’s capabilities found that the model certainly lives up to the hype. GPT-5 outperformed all other models we’ve tested on our datasets – by a lot. ### Comprehensive evaluation scores GPT-5’s weighted score from our comprehensive evaluations was **between 3465 and 3541** on different test runs – which is almost 200 points above OpenAI’s O3 model and Anthropic’s Sonnet 4, which were previously our highest scoring models. The maximum possible score is 3651. ![Benchmarking GPT-5: Why it’s a generational leap in reasoning](/content/assets/benchmarking-gpt-5-why-its-a-generational-leap-in-reasoning-asset.png) Full evaluation scores: * GPT-5: 3465–3541 * O3: 3288 * Sonnet-4: 3242 * Opus-4: 3170 **Takeaway:** While a 200 point or 5% increase might not seem significant, the way our tests work is that models initially rack up points finding low-hanging fruit like infinite loops and exposed secret keys. After a point, it then becomes progressively harder to get points since all the remaining points come from flagging much harder to find issues. GPT-5’s ability to get so many more points than other models, therefore, represents a significant leap forward in reasoning. ### Pass/fail scales We also give models a pass/fail score based on how many of the 300 error patterns in our dataset PRs the model was able to find. GPT-5 also achieved the highest success rate on this scale that we’ve ever seen at **254 to 259 out of 300.** Compare that to the performance of other models: ![Benchmarking GPT-5: Why it’s a generational leap in reasoning](/content/assets/benchmarking-gpt-5-why-its-a-generational-leap-in-reasoning-asset-2.png) * GPT-5: 254-259 * Sonnet-4: 212 * O3: 207 * Opus-4: 200 Since about 100 of the bottom PRs are found by all models, if we just look at the most difficult 200 error patterns, the numbers show even greater improvement with GPT-5 catching 78% of those error patterns and other models catching only 54% to 58%. ![Benchmarking GPT-5: Why it’s a generational leap in reasoning](/content/assets/benchmarking-gpt-5-why-its-a-generational-leap-in-reasoning-asset-3.png) * GPT-5: 157 * Sonnet-4: 117 * O3: 113 * Opus-4: 108 **Takeaway:** Similar to our comprehensive metric, the additional error patterns that GPT-5 was able to find are particularly hard for LLMs to spot, like concurrency bugs or inconsistent domain keys across environments, suggesting the model’s increased ability to reason. ## Hardest PRs test To stress-test each model, we curated 25 of the most difficult pull requests from our Golden PR Dataset. These PRs represent real-world bugs that span: * Concurrency issues (e.g. TOCTOU races, incorrect synchronization) * Object-oriented design flaws (e.g. virtual call pitfalls, refcount memory model violations) * Performance hazards (e.g. runaway cache growth, tight loop stalls) * Language-specific footguns (e.g. TypeScript misuses, C++ memory order subtleties) Each model was tested across three runs. Below is the **average pass rate** on this Hard 25 benchmark: ![Benchmarking GPT-5: Why it’s a generational leap in reasoning](/content/assets/benchmarking-gpt-5-why-its-a-generational-leap-in-reasoning-asset-4.png) ### Pass rate chart | **Model** | **Mean Pass Rate (%)** | | --- | --- | | Sonnet-4 | 26.7% | | Opus-4 | 33.3% | | O3 | 44.0% | | **GPT-5** | **77.3%** | **Takeaway**: GPT-5 shines where accuracy, contextual linkage, and depth matter most. It consistently delivers **the most complete, test-ready, and forward-compatible code review output** among all models we’ve tested to date. ## **How many kinds of bugs does GPT-5 actually catch?** To better understand ***what kinds*** of issues each model identifies—not just how many—our team reviewed every comment across a set of hard PRs and classified them into categories like **Concurrency**, **Security**, and **Object-Oriented Design**. We applied **deduplication** across models: if multiple models flagged the same core issue (even if phrased differently), it was counted only once per PR. This ensured we were measuring **issue coverage**, not comment verbosity. Then, for each model, we tallied what percentage of those unique issues it successfully caught. ### **Takeaway:** * **GPT-5 leads in almost every category**, identifying **over 60%** of concurrency, performance, and memory bugs — and an impressive **80%** of security issues. * **Security remains the most striking gap**: GPT-5 found 80% of security-related bugs, while the next best model (O3) found only 40%. * Even on basic concurrency and performance problems, GPT-5 consistently outperforms by 20-30 points. ![Benchmarking GPT-5: Why it’s a generational leap in reasoning](/content/assets/benchmarking-gpt-5-why-its-a-generational-leap-in-reasoning-asset-5.png) ## **Example: GPT-5 uncovers hidden concurrency risks missed by others** In this pull request, a subtle concurrency bug stemmed from a combination of **double-checked locking** and **unsafe access to a shared** `HashMap` in a singleton service class. While most models flagged the obvious thread-safety issue, GPT-5 delivered a comprehensive, production-ready fix—resolving not just the symptom, but the architectural flaws underneath. ### **The problem** The `OrderService` singleton used a `HashMap` to store orders, while concurrent updates were made from a fixed thread pool. This design lacked synchronization, leading to potential data corruption. On top of that, the singleton was initialized using a non-volatile static field—opening the door to **unsafe publication** and partially constructed objects. ### **GPT-5’s recommendations** GPT-5 went beyond the basic fix and stitched together a complete concurrency hardening plan: ### **1\. Replace the map with a thread-safe alternative** ```plaintext - private final Map orders = new HashMap<>(); + private final Map orders = new ConcurrentHashMap<>(); ``` ✅ GPT-5 also explained why: “Concurrent updates... are executed on a plain HashMap... not thread-safe and can lead to undefined behavior.” --- ### **2\. Fixed the broken singleton instantiation** ```plaintext - private static OrderService instance; + private static volatile OrderService instance; ``` Or optionally: ```plaintext private static class Holder { private static final OrderService INSTANCE = new OrderService(); } public static OrderService getInstance() { return Holder.INSTANCE; } ``` ✅ It flagged the classic memory visibility issue with double-checked locking and offered an alternate pattern to make construction thread-safe. --- ### **3\. Added a test reset hook to prevent state leakage** ```plaintext // Inside OrderService.java void clearAllForTest() { orders.clear(); } ``` ✅ This enables isolated, repeatable tests when working with a shared singleton across multiple test cases. --- ### **4\. Added timeouts to catch async test hangs** ```plaintext - future.get(); // Wait for completion + assertTimeoutPreemptively(Duration.ofSeconds(5), () -> future.get()); ``` ✅ GPT-5 proactively hardened the test suite by guarding against test flakiness in asynchronous flows. --- ### **What Sonnet-4 and Opus-4 missed** Both models correctly flagged the unsynchronized `HashMap` and replaced it with `ConcurrentHashMap`. However, neither delivered a complete or production-safe remediation: * ❌ **Singleton issues unresolved**: Sonnet-4 ignored the broken double-checked locking; Opus-4 mentioned it but skipped the actual fix (no `volatile`, no `holder` idiom). * ❌ **No test safety provisions**: GPT-5 introduced `clearAllForTest()` and timeout guards; Sonnet-4 and Opus-4 missed these entirely or only noted them passively. * ❌ **Lacked architectural context**: Neither model cross-referenced the broader codebase or justified changes with evidence. GPT-5 backed each fix with reasoning that traced across services, tests, and threading behavior. * ❌ **Limited scope**: Sonnet-4 made a single, surface-level fix. Opus-4 added some useful logging but missed the deeper structural risks GPT-5 fully addressed. --- ### **Why this matters** The real value of GPT-5’s review lies in its **depth and awareness**. It not only patched the visible race, but also: * Identified deeper architectural risks * Cross-referenced test reliability and code quality * Delivered a set of changes that are *safe to merge immediately* **This isn’t just a fix—it’s engineering insight.** GPT-5 showed how an AI reviewer can reason across system layers, suggest durable solutions, and help teams write safer code with less guesswork. ## **What’s new (and exciting) about GPT-5** ![Benchmarking GPT-5: Why it’s a generational leap in reasoning](/content/assets/benchmarking-gpt-5-why-its-a-generational-leap-in-reasoning-asset-6.png) Beyond metrics and the specific things our tests were evaluating, we found that GPT-5 exhibited new behavioral and reasoning patterns. * **Advanced contextual reasoning:** GPT-5 proactively planned multiple review steps ahead, showcasing expansive creative reasoning rather than strict input-bound logic. For example, GPT-5 demonstrated deep reasoning by connecting evidence across filles in our Concurrency oriented test focused on a ‘Check-then-act race condition’ scenario. It was the only model to detect risk of duplicate creation and introduced an atomic refund pattern grounded in the enum and test suite. * **Chain-of-thought reasoning via review threads**: In an Object-Oriented test focused on a Virtual call in constructor case, GPT-5 showed layered logic by first identifying a misused polymorphic override and then adjusting its recommendations based on its own earlier suggestions. This shows layered logic by identifying one thing and then showing additional reasoning on the issue later. * **Evidence-based diff justification**: In a Performance-focused test focused on Unbounded cache growth (no eviction) issue, GPT-5 identified architectural memory risks that other models missed, and backed its recommendation with diff context, usage patterns, and suggested safeguards. * **Forward-thinking suggestions**: In a Concurrency-related test focused on Incorrect sync primitive usage, GPT-5 not only patched the race but also suggested how to structure future additions, lock hierarchies, and test guardrails to prevent regressions. * **Granular, task-oriented recommendations:** Unlike previous models, GPT-5 detailed explicit follow-up tasks, creating actionable workflows within the review process itself. This makes the model much better for multi-step workflows. ## How we’re using GPT-5 in our AI code reviews We’re excited that GPT-5 represents a significant advancement in AI-powered code review, pushing the boundaries in detail, accuracy, and contextual reasoning. That’s why we’ll be using GPT-5 as the core reasoning model in our pipeline – starting today. We’re excited that it will be able to find more issues and create more in-depth, context-rich reviews. If you’ve never tried CodeRabbit, tried it previously, or are a current user, we’d love to hear how you think GPT-5 is improving your review quality and experience. ***Try our*** [***free 14-day trial today***](https://coderabbit.link/i72LrCC) ***to see the power of GPT-5 yourself.***

GitHub Copilot best practices: 10 tips & tricks that actually help

Ankur Tyagi — Wed, 23 Jul 2025 00:00:00 GMT

Copilot has quickly become a staple in the modern developer’s toolkit. Powered by OpenAI’s models, it offers AI-driven code suggestions based on what you’re writing — right in your editor. Used well, it can significantly boost productivity. Microsoft’s data suggests it may help developers [code up to 55% faster.](https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/) ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-inline-image-2.png) But here’s the catch: **Copilot isn’t a magic wand**. Left on autopilot, it can feel more like an eager junior dev making confident guesses than a reliable coding partner. The difference between a helpful Copilot and a frustrating one often comes down to how you use it — and whether you’ve built a workflow that plays to its strengths. In this article, we’ll walk through how Copilot fits into the broader [AI dev tool stack](https://www.coderabbit.ai/blog/2025-the-year-of-the-ai-dev-tool-tech-stack) and share practical GitHub Copilot tips and tricks for using it more effectively. These strategies are drawn from both our own experience and the thousands of developers using CodeRabbit’s AI code review platform. With the right approach, Copilot can go from a neat autocomplete toy to a genuinely valuable part of your daily development routine. ## GitHub Copilot tip 1: Play to Copilot’s strengths Not every coding task is equal in Copilot’s eyes. One of the most important GitHub Copilot best practices is to use it where it shines, not force it to create code where it doesn’t. Copilot excels at specific categories of tasks that can save you significant time. Copilot is especially good at… * Writing *repetitive code* * Generating *unit tests* * D*ebugging syntax issues* * E*xplaining code* * G*enerating regex patterns* These are areas where it has seen lots of examples and can confidently suggest solutions. For example, if you have a function and need to write several tedious unit tests for it, Copilot can draft them in seconds. Consider this simple function and tests:

def multiply(a, b):
"""Return the product of a and b."""
return a * b

Copilot can help create unit tests for the above function quickly: ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-inline-image.png) In a scenario like the above, Copilot generated the **TestMultiply** class almost entirely from a comment or prompt. It’s excellent for boilerplate code, repetitive patterns, and well-defined algorithms. On the flip side, Copilot is **not** a silver bullet for everything.It’s not designed to handle tasks unrelated to coding (don’t expect it to plan your database schema or design UI work) and won’t replace your problem-solving skills. Think of Copilot as a junior developer at your side. It’s fast and often right about everyday tasks, but you (the senior developer) are still in charge of decision-making and critical thinking. Use Copilot for the “heavy lifting” on mundane code and let it suggest solutions for routine problems, but always apply your judgment on whether to use those suggestions. That way, you’ll save time and reduce drudgery while keeping yourself focused on the challenging problems and design decisions. ## **GitHub Copilot tip 2:** Provide ample context (open files, set imports, etc.) A **hot tip for GitHub Copilot** is to open all the relevant files in your project when you’re coding a particular feature. That’s because Copilot works by looking at the context in your editor to predict what you might want next. The more relevant context you give it, the better the suggestions. For instance, if you’re implementing a function in **utils.py** that interacts with **models.py**, have both files open. Copilot will process all open tabs (often called “neighboring tabs”) to inform its suggestions. This broader view helps it understand your project structure and produce more accurate code. In fact, simply opening related files in VS Code or your IDE can significantly enhance Copilot’s completions by providing extra context for definitions and usages across your project. Similarly, explicitly set up your imports, includes, and dependencies before expecting the best suggestions. You know what libraries or frameworks you intend to use – tell Copilot by importing them at the top of your file. This gives Copilot a heads-up on what tools it should use. It’s often best to manually add the modules or packages (with specific versions, if needed) before asking Copilot to generate code using them. By doing so, you avoid Copilot defaulting to an outdated library or missing an import. ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-asset.png) **For example**, if you plan to use **pandas** in your code, write **import pandas as pd** yourself; then when you ask Copilot to manipulate a DataFrame, it will already know to use pandas and won’t attempt a pure-Python solution or an incorrect import. ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-asset-2.png) Also, be mindful of irrelevant context. Copilot’s window of attention is limited. If you have a lot of unrelated files open or leftover code in your editor, close or remove them when you switch tasks. Keeping only the pertinent files and context visible ensures Copilot isn’t “*distracted*” by code that doesn’t matter to your current goal. ## **GitHub Copilot tip 3:** Write descriptive comments and docstrings as prompts You understand prompt engineering when you’re directly calling an LLM. But did you know that there are some sneaky ways to prompt engineer in Copilot? One of the most effective GitHub Copilot tips is to guide the AI with natural language comments. Think of writing comments as a form of *prompt engineering*. Before you write the code, describe in plain English (or your preferred language) what you intend the code to do. **For example**, we want a function to sort a list of names case-insensitively. We might start with: ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-asset-3.png) The moment you write that comment and pause, Copilot will likely suggest the rest of the function (e.g., using sorted(names, key=str.lower)). A top-level comment at the start of a file, or a docstring/comment above a function, helps Copilot understand the overarching objective before diving into implementation details. This process is similar to giving a human colleague a quick overview of the task at hand, it sets the stage so the following code makes sense in context. When writing these comments, be clear and *specific* about the desired behavior. Mention any requirements or constraints. For a more complex example, suppose you need a function to format a person’s name as **"LASTNAME, Firstname"**. You could provide an example in the comment to clarify your intent: ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-asset-4.png) By including the example of input and output, you give Copilot a crystal-clear idea of what you want. Which is exactly the desired solution. The first comment was a prompt describing the goal and even provided a test case, and Copilot filled in the implementation. **Use this technique liberally.** Add a brief docstring or comment for each function describing *what* it should do (and how at a high level, if you have an approach in mind). Copilot can detect the comment syntax for your language and will often even help complete the comment if it recognizes a pattern (for example, it might suggest a template for a Python docstring). By writing specific, well-scoped comments before the code, you essentially “program” Copilot with your intent. **Remember the old saying**: garbage in, garbage out. If you feed Copilot an ambiguous comment like “# do something with data”, you’ll get ambiguous code. Instead, describe the task clearly – *“# Calculate the average value from a list of numbers, ignoring any nulls”* and watch Copilot more reliably produce the correct logic. ## **GitHub Copilot tip 4:** Use meaningful names for clarity You might hate being stickler about style but variable and function names are another form of context that Copilot relies on. A tip that might seem obvious but is often overlooked is to**give your functions and variables meaningful, descriptive names**. If you have a function named **foo()** or variable **data1**, Copilot has virtually no clue what you intend, beyond what it can infer from a possibly sparse usage context. In contrast, names like **calculate\_invoice\_total()** or **user\_email\_list** immediately convey intent to humans and the AI. In fact, Copilot’s suggestions will improve dramatically when your code is self-documenting. A function called **fetchData()** doesn’t mean much to Copilot (or to a coworker) compared to a function named **fetch\_airport\_list()** or **get\_user\_profile**. The latter gives far more hints of what the function should do. **For example**, consider these two scenarios: **Vague naming:**

# Determine if a user is eligible for promotion
def check(user, data):
# ...

With a function name like check and a parameter data, Copilot might struggle. “Check” could mean anything. Could it check a password or a value in data? Its suggestions might be generic or incorrect because it’s guessing your intent. **Descriptive naming:** ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-asset-5.png) The function name **is\_user\_promotable** clearly signals a boolean decision, and the parameters **user\_profile and promotion\_rules** indicate the data involved. Copilot can use this information to guess that you might iterate over rules, check user attributes, etc., and its completion will align with that logic. Adopting clear naming conventions isn’t just a general coding best practice for developers. It’s a GitHub Copilot best practice, too, because Copilot can only infer intent from what it sees. If what it sees are meaningful identifiers and not cryptic ones, it will return far more relevant code. This tip also pays dividends for code maintainability – since you’ll get better AI suggestions *and* cleaner code for your team. ## **GitHub Copilot tip 5:** Pair Copilot with CodeRabbit for AI-assisted code reviews While Copilot is fantastic during the coding phase, what about after you’ve written your code? Enter CodeRabbit, an AI-powered code review developer tool that complements Copilot in the development workflow. CodeRabbit acts like an AI “pair reviewer,” scanning your code (either in the [IDE](https://www.coderabbit.ai/ide) or on your [Git platform](https://docs.coderabbit.ai/platforms/github-com)) and providing feedback and suggestions for improvement. ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-asset-6.png) We’ve found that using Copilot and CodeRabbit together creates a powerful feedback loop: Copilot helps you generate code quickly and CodeRabbit helps ensure that code meets quality standards before it gets merged. * You don’t get the developer who wrote the code to review it, so why get the same AI system to do so? * An AI code reviewer also allows you to standardize your quality gate if your team is using multiple AI coding agents – as so many teams are these days. * Finally, purpose-built AI coding agents like CodeRabbit do a more thorough job and have more features that means the average user is able to find 50% more bugs in half the time they’d typically spend on a code review. CodeRabbit integrates into VS Code and pull requests on platforms like GitHub. In your IDE, you can invoke CodeRabbit to review the file or the diff you’re working on. It will directly add AI-powered inline review comments in the code, pointing out potential issues, much like a [human reviewer](https://www.devtoolsacademy.com/blog/ai-code-reviewers-vs-human-code-reviewers/) would. ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-asset-7.png) **For example**, CodeRabbit might flag that your function lacks error handling for a specific edge case or suggest a more appropriate HTTP status code. On GitHub or GitLab, CodeRabbit can automatically comment on PRs with its findings, saving human reviewers time by catching obvious problems first. It also provides line-by-line code reviews, highlighting possible bugs, code smells, style issues, or even missing unit tests. ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-asset-8.png) ### How best to use Copilot and CodeRabbit together Think of Copilot and CodeRabbit as two halves of a complete AI-assisted development cycle. You use Copilot while writing code to speed up implementation. Then you use CodeRabbit to review that code and catch anything Copilot (or you) might have missed. * Copilot might generate a solution that works but isn’t optimal and CodeRabbit could point out a performance issue or a more idiomatic approach. * Copilot might not know your project’s specific coding standards, but CodeRabbit can enforce them during review. Perhaps your team prefers format() over f-strings, etc.. CodeRabbit can comment on that. * Copilot might help you quickly whip up a new API endpoint, CodeRabbit could then run and immediately warn, *“Hey, you didn’t handle the case where this input is null,”* or *“This SQL query might not be parameterized.”* You can address those before your human colleagues even look at the code. Essentially, Copilot gets you to a working draft faster, and CodeRabbit gives you confidence to ship it by auditing the code. It’s like having an AI pair programmer and an AI code auditor working together. In the context of a [complete AI dev tool stack](https://www.coderabbit.ai/blog/2025-the-year-of-the-ai-dev-tool-tech-stack), Copilot and CodeRabbit cover a lot: Copilot for coding, CodeRabbit for review, and you might even use other AI tools for testing or security. ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-asset-9.png) To get started, [you can install CodeRabbit’s IDE extension](https://marketplace.visualstudio.com/items?itemName=CodeRabbit.coderabbit-vscode) or add it to your GitHub repository as a [GitHub App from the marketplace](https://coderabbit.link/j4daQIo). We highly recommend this for teams and there’s even a [14-day trial.](https://coderabbit.link/ZboRIZI) ## **GitHub Copilot tip 6:** Be specific and provide examples in prompts When it comes to guiding an AI model, specificity is king. If you’re asking Copilot to write code to transform data, consider providing a short example of the data format in a comment or docstring. If you want a function to calculate something, state the formula or an example scenario in natural language. Copilot’s underlying model is essentially trying to predict what a knowledgeable developer would write next. If your prompt (context + comments) is vague, the model must guess and may go wrong. But if you spell out details and even including sample inputs and outputs if possible, that helps For instance, suppose you need to parse a log line like **"2025-06-02 09:00:00 - ERROR - failed to connect"**. Instead of just writing **\# parse log line**, you could write: ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-asset-10.png) This specific prompt gives Copilot a clear blueprint: it knows the log format and the desired output types. With the example shown, the chances of Copilot writing a correct **parse\_log\_entry** implementation (splitting by " - ", parsing the date with **Datetime, strptime**, etc.) are much higher. Without the example, Copilot might misidentify the format or split incorrectly. When prompting Copilot for non-trivial code, spell out the details. If a function has constraints (e.g. “input can be null” or “assume list is sorted”), mention them. If there’s a particular approach you want (e.g. “use binary search” or “use recursion”), hint at it in your comment. And if possible, provide a quick example. The model will take these as strong cues and align its suggestions accordingly. ## **GitHub Copilot tip 7:** Break complex tasks into smaller steps Copilot works best when it’s dealing with a focused, well-defined task. If you ask it to do too much at once, you might get a muddled or incomplete answer. A great strategy is to break down big problems into bite-sized pieces and tackle them one by one with Copilot’s help. **For example**, imagine you need to implement a complex algorithm. Instead of prompting Copilot to write the whole thing in one go (which might result in a long, confusing blob of code), start by outlining the high-level steps as comments or pseudocode. You might write a few lines of comments or stub functions and then let Copilot fill in each part. Generate code incrementally, rather than all at once. This approach makes Copilot’s job easier (each step has more specific context). That makes it easier for you to review and trust the code at each step. Let’s say you’re building a small command-line program. First, prompt Copilot to parse command-line arguments, then separately prompt it to implement the business logic, then prompt it to handle output. You can check Copilot’s work at each stage by breaking the flow. ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-asset-11.png) Think of Copilot as participating in a step-by-step refinement of your code. So, don’t try to have Copilot write an entire module in one shot. Instead, have it write one function at a time, or even one logical block at a time, especially if the logic is intricate. By decomposing tasks, you also naturally create opportunities to review each piece. This incremental approach leads to better quality code and fewer surprises. It’s much easier to debug a smaller Copilot suggestion than the 50-line monolith it spit out because you asked a very broad question. ## **GitHub Copilot tip 8:** Leverage Copilot Chat vs inline completions wisely **GitHub Copilot has two primary flavors –** traditional inline code completion and the newer [Copilot Chat](https://docs.github.com/en/copilot/using-github-copilot/copilot-chat) (an interactive chat interface available in VS Code, Visual Studio, and other environments). Knowing when to use Copilot Chat vs when to rely on inline suggestions can make a big difference in your workflow. It’s one of those subtle hot tips for GitHub Copilot that can transform how you approach a problem. **Inline code completions** (the original Copilot experience) are best for: * **In-the-flow coding assistance:** When you’re writing code and want Copilot to suggest the next line or block as you type. This works great for completing a small algorithm, filling in a loop, or writing boilerplate in place. * **Filling in repetitive code or simple patterns:** For example, generating a quick data class, an API call, or the next cases in a series of if/elif conditions. The inline suggestions excel at continuing your current context. * **Generating code from a commented intent:** as we’ve seen, if you write a comment # do X, the inline completion often does X immediately in code form. On the other hand, Copilot Chat is more powerful when you need more interaction or have questions about your code: * **Explaining or analyzing code:** You can ask Copilot Chat *“What does this function do?”* or *“Why am I getting a KeyError here?”* and get a natural language answer. The chat can act like a super-smart rubber ducky for debugging. * **Larger code generation tasks with iteration:** If you want to generate a sizable chunk of code (say a whole function or class) and then refine it, Copilot Chat is ideal. You might ask it to write the code, then say, “now optimize this” or “can you refactor that part using a dictionary instead of if-else?” This back-and-forth is something inline suggestions can’t do easily. * **Using personas or specific commands:** Copilot Chat has a concept of *keywords and skills* (and allows system-level instructions like “Act as a senior developer…”) which you can use to influence its style or thoroughness. For instance, you could instruct it to be security-conscious when writing the code. To illustrate, if I have a piece of code and I’m not sure it’s efficient, I might use Copilot Chat: *“Explain the complexity of this function. How can I improve it?”* Copilot Chat might identify the bottleneck and even suggest a more efficient approach. ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-asset-12.png) On the other hand, if I just need the next few lines of a loop, inline completion is faster. I can just hit Tab and keep coding. **Tip:** If you have access to both, don’t forget you can use them together. Maybe start writing a test function (inline completion helps you fill out test cases), then switch to Chat to ask Copilot to generate some additional tests or explain a failing test. Each tool has its sweet spot. ## **GitHub Copilot tip 9:** Cycle through suggestions and refine your prompts By default, Copilot might show you one suggestion – the most likely completion – for your prompt or code context. But what if that suggestion isn’t what you want? Many users forget that Copilot usually has **multiple suggestions** under the hood. Don’t settle for the first thing it offers if it’s not quite right. A GitHub Copilot tip that’s helped me is to use the keyboard shortcuts (or the Copilot panel) to cycle through alternative suggestions. ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-asset-13.png) There might be a gem in suggestion #2 or #3 that better fits your needs than suggestion #1. ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-asset-14.png) Additionally, you can open the Copilot sidebar (or the full chat interface, if available) to explicitly ask for more options. In some IDE setups, hitting a special shortcut (like Ctrl+Enter in VS Code with Copilot Chat enabled) will even reveal multiple completions at once. Scanning through a few options can save you time you’d otherwise spend editing a less ideal suggestion. It’s like getting a second and third opinion from the AI. ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-asset-15.png) If none of the suggestions look good, that’s a signal to refine your prompt or add more context. Perhaps your comment was too short or ambiguous. Try rephrasing it or adding another detail and then trigger Copilot again. The model’s output can vary greatly with slight changes in how you ask. For instance, if the **#sort list** didn’t give the desired result, then **#sort the list of names in alphabetical order** might produce a better suggestion. Another trick is giving feedback to Copilot. If you’re using Copilot Chat or the sidebar, you might have thumbs-up/down buttons you can press to give feedback on suggestions. While this won’t instantly change the current suggestion, it does send feedback to improve the model’s future behavior through reinforcement learning. And in chat mode, you can directly say, “No, that’s not what I meant. I actually want X,” and the AI will try again. ## **GitHub Copilot tip 10:** Review, test, and verify Copilot’s output Copilot can generate code that looks perfect at first glance, but remember, it’s not guaranteed to be 100% correct or optimal. Always review and test the suggestions before integrating them into your codebase. This tip cannot be stressed enough. Copilot may introduce bugs, security issues, or logically wrong code if the prompt is misunderstood. You, the developer, are the last line of defense to ensure quality. To review Copilot’s output, first, read the code carefully and make sure you understand it. If Copilot suggests a complex algorithm or some math you’re unsure about, ask Copilot (via Chat) or use your knowledge to break down what it’s doing. You can ask Copilot Chat to explain the suggested code in plain language as a helpful Copilot trick. Often, I’ll paste a large suggestion into the chat and prompt: “Explain what this code does.” You can immediately catch any assumptions or errors if the explanation reveals any assumptions or errors. ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-asset-16.png) Next, consider edge cases and correctness. * Does the code handle empty inputs? * What about error conditions? * If something looks fishy (like a potential off-by-one error or an unbounded recursion), address it or prompt Copilot to fix it. Security and style are also important. If your prompt didn't specify otherwise, Copilot might use a deprecated function or an insecure approach. Always double-check things like SQL queries (are they parameterized to prevent injection?), file operations (are files closed properly?), and any cryptography or authentication code it writes (does it follow best practices?). **Linting and static analysis tools are your friends here**. Run your linters or code formatters on Copilot’s code to catch style issues, and use any security scanners (like Snyk or CodeQL) if applicable to flag vulnerabilities. Finally, remember that Copilot might occasionally produce code that is oddly similar to public examples (especially for very common algorithms). It’s rare, but if you’re working on a closed-source project and have strict license requirements, be mindful of this. You can configure Copilot to avoid suggestions that match public code, if needed. ![GitHub Copilot best practices: 10 tips & tricks that actually help](/content/assets/github-copilot-best-practices-10-tips-and-tricks-that-actually-help-asset-17.png) ## Now, it’s time to use these hot tips for GitHub Copilot! GitHub Copilot is a game-changer for developers but like any powerful tool, it yields the best results when used skillfully. We’ve covered our top 10 hot tips for GitHub Copilot – from crafting great prompts and leveraging context to integrating Copilot with an AI reviewer like CodeRabbit. By implementing these GitHub Copilot best practices, you’ll find Copilot becomes much more helpful. It can handle the boilerplate and suggest clever solutions, allowing you to focus on higher-level thinking and problem-solving. Also, don’t forget that Copilot and the [surrounding AI ecosystem](https://www.coderabbit.ai/blog/2025-the-year-of-the-ai-dev-tool-tech-stack) are evolving rapidly and new features (like the CLI tool, vision-based Copilot, etc.) are coming out regularly. Stay curious and keep experimenting with how you use it. Perhaps you’ll discover new GitHub Copilot tips and tricks beyond the ten we’ve shared. Or we’ll just write another article. ***Interested in using CodeRabbit with Copilot? Start your*** [***free 14-day trial.***](https://coderabbit.link/ZboRIZI)

The art and science of context engineering for AI code reviews

Amitosh Swain — Wed, 16 Jul 2025 00:00:00 GMT

The difference between a mediocre and exceptional AI agent comes down to one thing: *context*. Context engineering has recently become a buzzword. In late June, [Shopify CEO Tobi Lutke](https://x.com/tobi/status/1935533422589399127) tweeted about it and [Andrej Karpathy](https://x.com/karpathy/status/1937902205765607626) chimed in to point out that good context engineering is what sets AI apps apart. ![The art and science of context engineering for AI code reviews](/content/assets/the-art-and-science-of-context-engineering-inline-image.jpg) CodeRabbit is a great example of the difference context makes. Every day, we perform tens of thousands of code reviews — either on our users’ pull requests or in their IDEs. Each review comment that CodeRabbit makes is the result of a carefully engineered non-linear review pipeline that pulls in contextual data, related to code being reviewed, from dozens of sources and then runs verification agents to check again that every suggestion we share makes sense within the context of both the PR we’re reviewing and the greater codebase. Context engineering is the difference between an AI code review tool that merely pattern-matches against generic coding standards and one that deeply understands your project's specific architecture, patterns, and goals – and can actually add value to your code review. ## The nature of context engineering in code reviews ![The art and science of context engineering for AI code reviews](/content/assets/shared-shared-image.png) We break down the context in which CodeRabbit operates into three distinct parts: 1. **Intent**: What the developer or team aims to achieve with the code changes, including the purpose of a pull request, the problems they are trying to solve, and the intended outcomes. 2. **Environment**: The current technical state of the system, including file relationships, code dependencies, project structure, and existing patterns. 3. **Conversation:** The rest of the regular stuff that goes into a multi-turn LLM call, i.e., your chat messages, tool call responses, etc. When these elements are appropriately balanced and presented to an AI system, the result is an intelligent code reviewer that catches not just syntactic issues but also architectural inconsistencies, potential performance bottlenecks and opportunities for higher-level design improvements. ## Finding the context engineering sweet spot ![The art and science of context engineering for AI code reviews](/content/assets/the-art-and-science-of-context-engineering-inline-image.png) Creating the proper context for AI-powered code review involves navigating several challenges. Here are three challenges that make context engineering particularly difficult. ### 1\. The Goldilocks problem for AI agents * **Too little context** leads to "hallucinations"—where the AI makes assumptions about missing information, often incorrectly. * **Too much irrelevant context** dilutes the signal, causing the AI to focus on unimportant aspects or become overwhelmed with information. * **Just the proper context** provides precisely what the AI Agent needs for accurate insights without *noise*. ### 2\. Token-by-token processing Unlike humans, who can quickly scan a document and mentally prioritise important sections, AI models process information token-by-token, giving equal weight to each piece of text. If we put all the code changes from a PR in the prompt, the AI may latch on to something insignificant and skip major issues. Curating the context is important. You need to prioritise important changes and discard the unimportant ones. ### 3\. Context window limitations Even the most advanced AI models have finite context windows, the maximum amount of text they can process at once. This limitation makes strategic context selection critical, especially for large codebases or complex pull requests. ## The CodeRabbit approach to context engineering ![The art and science of context engineering for AI code reviews](/content/assets/the-art-and-science-of-context-engineering-asset.png) At CodeRabbit, we've developed a multi-layered approach to context preparation that addresses these challenges and delivers consistently high-quality code reviews. Our system employs a sophisticated, non-linear pipeline designed to gather, filter, and structure context in ways that maximise AI comprehension. The diagram above lists just some of the dozens of sources of context we draw on in our context preparation process. ### Intelligent repository and PR information collection ![The art and science of context engineering for AI code reviews](/content/assets/the-art-and-science-of-context-engineering-asset-2.png) Our context preparation begins with extracting the most relevant information about the pull request itself: * **Metadata**: We collect essential data like PR title, description, and affected commit range to determine the "why" behind code changes. * **Differential analysis**: For incremental reviews, we calculate exact changes since the last review, ensuring the agent focuses only on what's new or modified. * **Path filtering**: Our system distinguishes between meaningful code changes and [ancillary files](https://docs.coderabbit.ai/guides/review-instructions/#default-blocked-paths) (like generated assets or dependencies), focusing the AI's attention on what truly matters. ### Knowledge integration from multiple sources ![The art and science of context engineering for AI code reviews](/content/assets/the-art-and-science-of-context-engineering-asset-3.png) A great code review requires more than examining the current changes in isolation. Next, we work on understanding the broader technical and business context: * **Historical learnings**: We employ a vector database to store our agent’s learnings from past reviews, allowing the system to recall relevant feedback patterns and user preferences so it can structure review comments with these in mind. * **PR intent analysis**: We analyse PR descriptions and related issues to extract the underlying objectives of changes, ensuring CodeRabbit's review aligns with developer goals. * **Code Graph Analysis**: We then construct a graph representation of code dependencies to help the AI understand how files interrelate, enabling reviews that consider architectural impact. ### Strategic context assembly ![The art and science of context engineering for AI code reviews](/content/assets/the-art-and-science-of-context-engineering-asset-4.png) Once we have gathered all the raw information required for reviewing, we optimize how the prompt is packaged for the AI agent. ### Prompt engineering ![The art and science of context engineering for AI code reviews](/content/assets/the-art-and-science-of-context-engineering-asset-5.png) The next stage of our review pipeline involves crafting the perfect instructions for the AI. We average a 1:1 ratio of code to context in our prompts which shows how important context is: * **Level-appropriate prompts**: We adjust the review depth based on file complexity and importance, ranging from basic checks to in-depth architectural analysis. For different complexity levels, we use different prompts and models. * **Structured review guidelines**: Clear instructions help the AI Agent focus on the most valuable types of feedback for each specific situation based on our own historical data on helpful review comments * **Context Enrichment**: The prompts include relevant project coding standards, patterns, and historical insights that guide the AI toward company-specific best practices. * **Context Selection**: We perform a final pass of the context with results of the previous agents, which did the context preparation to cut noise. ### Verification agents ![The art and science of context engineering for AI code reviews](/content/assets/the-art-and-science-of-context-engineering-asset-6.png) The final target of our review process is our verification system which is an AI-powered quality assurance layer that automatically validates and improves review comments. It’s activated when the AI reviewer needs to double-check its findings. ## Impact of context engineering on review quality ![The art and science of context engineering for AI code reviews](/content/assets/the-art-and-science-of-context-engineering-asset-7.png) The sophistication of our context engineering preparation pipeline directly translates to tangible benefits in review quality, including: ### Reduced false positives By providing the AI with the proper context, we dramatically reduce irrelevant or incorrect suggestions that waste developer time like, for example, changes to function calls that don’t align with the team’s coding standards. The system understands project-specific conventions and avoids flagging intentional patterns as issues. ### Deeper architectural insights With more knowledge of code relationships and project structure, CodeRabbit can identify architectural issues that simple linting tools or pattern matching will miss. For example, many of our customers recount how CodeRabbit is able to flag when changes in a PR will affect other dependencies in their codebase. ### Consistent application of best practices By incorporating historical learnings and team-specific knowledge, we consistently apply coding standards and best practices across all reviews. We continue to make it easier for teams to share their coding guidelines – including recently enabling the ability to [import coding guidelines](https://www.coderabbit.ai/blog/code-guidelines-bring-your-coding-rules-to-coderabbit?) from your favorite coding agent. ### Enhanced learning over time Our approach enables the system to improve with each review, building a growing knowledge base of project-specific insights that make future reviews even more valuable. ## The importance of good context engineering Context is not merely a technical requirement for LLMs, but a requirement for effective AI Agents. By thoughtfully gathering, filtering, structuring, and presenting the context, CodeRabbit doesn't just review code. Instead, it understands code in its full complexity so that it can provide insights that make developers more productive, code more robust, and teams more effective. This is increasingly important as AI coding agents currently tend to generate a significant amount of AI slop. This is only the beginning of context engineering for AI code reviews – we are always refining our approach to improve review quality. With model capabilities constantly expanding, in the future, we’ll be able to do much more. ***Interested in seeing how context makes a difference in our code reviews?*** [***Start your 14-day trial!.***](https://coderabbit.link/fOmVdcW)

How we built dashboards with a micro-frontend and Grafana

Gurinder Singh — Mon, 02 Jun 2025 00:00:00 GMT

Modern dev teams rely on data. Without analytics, how would you know if your team is improving its deployment velocity and code quality over time? At CodeRabbit, we want teams to have the data that matters when it comes to tracking their performance. That’s why we created a unique integration with [Grafana](https://grafana.com/), a leading analytics platform, that provides greater visibility into your organization’s code review metrics through interactive dashboards directly in the CodeRabbit UI. [CodeRabbit](http://www.coderabbit.ai), an AI-powered code reviewer, can be seamlessly installed on your Git platform to review pull requests and deliver actionable insights. By embedding Grafana as a micro-frontend, we’ve made it easier for teams to understand the impact of code reviews on their organization. In this post, we’ll cover why we decided to integrate dashboards in our application via a micro-frontend framework and how we built our Single SPA micro-frontend with the help of [Qiankun](https://qiankun.umijs.org/). ## Why use Grafana as a Single SPA micro-frontend? When it comes to showing analytics in a single-page application, developers often face challenges. Building custom dashboards from scratch requires significant coding effort, and maintaining that code becomes a burden. Adding new metrics or dashboard panels typically involves creating new APIs and modifying frontend code, making the process slow and cumbersome. To address these challenges, we decided to leverage Grafana as a Single SPA micro-frontend. This decision allowed us to streamline the addition of new dashboards while managing Grafana separately and deploying updates with ease. By doing so, we reduced development overhead and improved scalability. ## Adding Grafana as a Single SPA micro-frontend with Qiankun Grafana offers robust data visualization capabilities. To integrate it seamlessly into our UI, we [forked Grafana](https://github.com/coderabbitai/grafana), customized its dashboard page, and implemented it using [Qiankun](https://qiankun.umijs.org/). This micro-frontend architecture enabled us to retain the native Grafana experience while tailoring the interface to align with CodeRabbit’s needs. Micro-frontend architectures are particularly useful when integrating multiple JavaScript frontend applications, even those built with different frameworks. After exploring various micro-frontend framework options, we chose the [Qiankun](https://qiankun.umijs.org/) micro-frontend library to implement this architecture. Qiankun, built on top of the [single-spa](https://single-spa.js.org/) micro-frontend framework, provides a simple API that makes it easy to manage micro-frontend architectures. ## Using the Qiankun micro-frontend framework to overcome architectural challenges Most micro-frontends are mounted in the main app based on specific routes. However, our use case required dashboards to be displayed dynamically across multiple routes and on tab changes. To achieve this, we used Qiankun’s loadMicroApp API, which implements the single-spa parcel API under the hood. This approach eliminated common limitations of micro-frontend architectures. To improve performance and reduce load time, we used caching in our Grafana proxy server. ## Securing our Grafana micro-frontend with authentication Authenticating Grafana using our UI credentials was a critical challenge. Typically, Grafana relies on an API key for access. To handle this, we created a proxy server that first authenticates a CodeRabbit user and then forwards requests to Grafana. While this setup worked initially, it exposed a potential vulnerability: Grafana allowed any query to be sent in the request, posing a risk of data leaks. To mitigate this, we added a validation layer that: 1. Ensures only pre-defined queries are allowed. 2. Scopes requests to specific organizations and restricts access to queries explicitly associated with dashboards. This solution secured the Grafana micro-frontend integration by preventing unauthorized data access and ensuring a robust implementation. ![How we built dashboards with a micro-frontend & Grafana](/content/assets/simplify-dashboards-with-a-micro-frontend-and-grafana-asset.png) The flow diagram above shows a micro-frontend architecture overview. The CodeRabbit UI mounts the Grafana dashboard using Qiankun by accessing a public micro-frontend endpoint that exports component lifecycle functions. Once mounted, all API calls to Grafana are routed through our authentication service for secure access. ## Our use case for dashboards ![How we built dashboards with a micro-frontend & Grafana](/content/assets/simplify-dashboards-with-a-micro-frontend-and-grafana-asset-2.png) At CodeRabbit, our goal is to demonstrate the value our product adds to an organization by providing AI-driven code reviews. To achieve this, we collect and analyze data stored in our data layer and use it to create insightful panels that tell the story of CodeRabbit's impact on our customers’ development cycle. ![How we built dashboards with a micro-frontend & Grafana](/content/assets/simplify-dashboards-with-a-micro-frontend-and-grafana-asset-3.png) The dashboards include metrics such as the average number of pull requests (PRs) reviewed per day and the total number of reviews conducted by CodeRabbit. Additionally, they showcase the various language-specific tools leveraged to fine-tune reviews, offering deeper insights into how CodeRabbit adapts to diverse development environments. These panels also highlight key contributions, such as the number of comments, suggestions, and chat conversations facilitated by CodeRabbit to improve code quality. ![How we built dashboards with a micro-frontend & Grafana](/content/assets/simplify-dashboards-with-a-micro-frontend-and-grafana-asset-4.png) By integrating Grafana as a Single SPA micro-frontend, we were able to design and serve these visualizations seamlessly within the CodeRabbit UI while reducing the amount of development and maintenance work. ## Benefits of Single SPA micro-frontend architecture Integrating Grafana using a Single SPA micro-frontend architecture has transformed how we deliver analytics to our users. Since Grafana is a separate service, we just have to modify its dashboards and provision the changes using Grafana Dashboard provisioning. Using Grafana as a Qiankun and Single SPA micro-frontend in CodeRabbit has allowed us to deliver a powerful, secure, and dynamic analytics experience for our users. Looking ahead, we plan to expand our dashboard offerings and explore additional Grafana plugins to provide even more insightful analytics. For developers looking to streamline analytics integration, we highly recommend exploring Grafana as a micro-frontend—it’s a game-changer for simplifying dashboard management and enhancing scalability. Interested in trying out CodeRabbit? [Get a free 14-day trial.](https://app.coderabbit.ai/login?free-trial=) Want to join our team? Check out our [Careers page!](https://coderabbit.link/63iJMZY)

Pipeline AI vs agentic AI for code reviews: let the model reason — within reason

David Loker — Thu, 29 May 2025 00:00:00 GMT

AI has changed what code reviews can be. We’ve gone from static rules and regex-based linters to systems that can actually read a diff and respond with feedback that resembles what a senior engineer might say. That’s real progress. But as companies like CodeRabbit create production-grade systems for code reviews or for other developer-focused tools, we all face a core architectural question: **Do you give the AI autonomy to plan and act like an agent? Or do you structure the process as a predictable AI pipeline?** This choice affects more than just implementation. It shapes how fast your system runs, how much developers trust it, how you debug it when it breaks, and what it takes to maintain it long-term. And while the architecture matters, it's not the end goal. These are just different ways of trying to answer the same underlying question — *How do we give the model everything it needs (and nothing more) to deliver the best code review possible?* That’s the real challenge. Not "agentic AI" vs. "pipeline AI." Just building the best possible tool for the people who use it. We’ll come back to that. But first, let’s define the two camps. ## **AI architectural patterns: Agentic AI vs. pipeline AI** ### **Agentic AI systems** ![Pipeline AI vs agentic AI for code reviews: Let the model reason —](/content/assets/pipeline-ai-vs-agentic-ai-for-code-reviews-let-the-model-reason-within-reason-inline-image.png) In an agentic architecture, the model isn’t locked into a single prompt. It’s allowed to think step-by-step, make decisions, and use tools as it goes. Often this means: * Planning a course of action * Calling a tool (e.g. grep, a static analyzer, test runner) * Observing the output * Deciding what to do next This approach — often referred to as **ReAct (Reason + Act)** — is one of several reasoning patterns used to guide agent behavior. It shows up across a range of modern systems and research prototypes, but the core idea is the same: the model can reason, act, observe, and repeat — using external tools and memory to enrich its output. That flexibility is incredibly promising. It’s also incredibly hard to get right. ### **Pipeline AI systems** ![Pipeline AI vs agentic AI for code reviews: Let the model reason —](/content/assets/pipeline-ai-vs-agentic-ai-for-code-reviews-let-the-model-reason-within-reason-inline-image-2.png) Pipeline AI-based systems take a more deterministic approach. You define a sequence: 1. Prepare inputs (e.g. diff, relevant file slices, issue text) 2. Run pre-processing (e.g. static analysis, code search) 3. Call the model with a crafted prompt 4. Post-process the output into review comments This approach is predictable, fast, and easy to test. It’s also easier to integrate into CI workflows, where speed and reproducibility matter. Many tools use a pipeline AI backbone as their foundation, however, most modern implementations also incorporate elements of agentic behavior. They may dynamically adjust prompts, use retrieval strategies, or support interactive review flows. They aren’t fully agentic, but they aren’t rigidly linear either. Which brings us to the reality most teams face: you don’t have to pick a side. Most real-world systems live somewhere in the middle — not for philosophical reasons, but because that’s what it takes to ship something reliable, adaptable, and useful. ## **Hybrid AI systems: A spectrum, not a binary** ![Pipeline AI vs agentic AI for code reviews: Let the model reason —](/content/assets/pipeline-ai-vs-agentic-ai-for-code-reviews-let-the-model-reason-within-reason-asset.png) In practice, many real-world systems don’t land fully in either the agentic AI or pipeline AI camp. They blend elements of both — taking the structure and reliability of pipelines, and layering in tool use, learned behavior, or context enrichment strategies that are often associated with agents. [**CodeRabbit**](https://cloud.google.com/blog/products/ai-machine-learning/how-coderabbit-built-its-ai-code-review-agent-with-google-cloud-run) is a good example of this kind of hybrid AI architecture. [GitHub Copilot PR Reviews](https://github.blog/changelog/2025-04-04-copilot-code-review-now-generally-available) also falls into this category. While their interfaces and goals differ, they share similar DNA — blending structured inputs with retrieval, static analysis, and interactive flows. We go deeper into CodeRabbit’s AI pipeline and enrichment strategy in the next section, but in short: it blends the determinism and predictability of pipelines with dynamic, learned behavior and targeted context augmentation — sitting squarely between the two paradigms. Hybrid AI systems like this sit along a spectrum — and that's the point. You don’t have to go all-in on one paradigm. You just have to solve for what matters: helping your users make better decisions, faster, with fewer surprises. Hybrid systems aim to balance the pros and cons of both agentic and pipeline systems by finding a balance between the two. Striking the right balance can also be difficult to achieve, with some experimentation required. This added control and flexibility can increase the cost of development and maintenance. ## **Tradeoffs between AI architecture patterns** ![Pipeline AI vs agentic AI for code reviews: Let the model reason —](/content/assets/pipeline-ai-vs-agentic-ai-for-code-reviews-let-the-model-reason-within-reason-asset-2.png) | **Dimension** | **Agentic systems** | **Pipeline systems** | | --- | --- | --- | | **Latency** | Multi-step, often slower | Fast, predictable | | **Tool Use** | Dynamic and adaptive | Static and consistent | | **Trust** | Harder to test, less predictable | Easier to debug and validate | | **Context Handling** | On-demand, but error-prone | Predefined and controlled | | **Workflow Fit** | Interactive tools | CI/CD and production PR reviews | Agentic AI systems offer flexibility — but flexibility is a double-edged sword. They can fetch exactly what’s needed… or fetch everything and drown in noise. They can reason step-by-step… or loop forever. You need good defaults, good tools, and often, some level of hard constraint. Pipelines, by contrast, are stable. You get speed, control, and a well-bounded behavior space. But they can be rigid. If the context isn’t there at the start, the model can’t do much about it. That’s the tradeoff. And that’s what most of us are doing here — not debating abstractions, but working to build the best damn tool we can. For ourselves. For our teams. For the developers who need to ship something today. The AI architecture pattern you use is just a means to an end. The real work — and the real leverage — lies somewhere else. ## **AI context is the real bottleneck: Why autonomy needs structure** ### **More context isn’t always better** In AI code review, we spend a lot of time debating architecture — agentic AI vs. pipeline AI — but the real performance bottleneck is often upstream: **what context we give the model**. There’s a common assumption: *If we just add more AI context — more code, more metadata, more analysis — the model will perform better.* But that’s not how it works. * Too much irrelevant input overwhelms the model ([Secure Code Review at Scale](https://github.com/247arjun/ai-secure-code-review/blob/main/Automated%20Secure%20Code%20Review%20at%20Scale%20Using%20Static%20Analysis%20and%20Generative%20AI.md#:~:text=Another%20lesson%20learned%20is%20that,be%20a%20very%20subjective%20measure)) * Prompt noise leads to muddled reasoning and false positives ([Secure Code Review at Scale](https://github.com/247arjun/ai-secure-code-review/blob/main/Automated%20Secure%20Code%20Review%20at%20Scale%20Using%20Static%20Analysis%20and%20Generative%20AI.md#:~:text=LLMs%20tend%20to%20hallucinate%20far%20less%20when%20analyzing%20a%20specific%20code%20snippet%20or%20diff)) * Even high-quality tools can generate low-quality AI context if used indiscriminately ([Anthropic Case Study](https://www.anthropic.com/customers/coderabbit#:~:text=assurance%20processes%20requiring%20multiple%20specialized,dependent%20vulnerabilities)) More isn’t better. **Better is better.** ### **Agent autonomy sounds great — but struggles in practice** Agentic systems promise flexibility: let the model decide what it needs, when it needs it, and fetch context accordingly. In theory, this is ideal. In practice, it’s messy. Common failure patterns: * Tool overuse — agents calling everything, just in case ([DevTools Academy)](https://www.devtoolsacademy.com/blog/coderabbit-vs-others-ai-code-review-tools/#:~:text=unnecessary%20things%20$there%20were%20cases%20of%20early%20AutoGPT%20agents%20doing%20redundant%20web%20searches$) * Redundant or noisy fetches that dilute the prompt ([Prompt Engineering Guide](https://www.promptingguide.ai/research/llm-agents#:~:text=overhead%20and%20sometimes%20unpredictability%20if%20the%20agent's%20prompt%20isn't%20tightly%20controlled)) * No clear reward signal to distinguish helpful context from useless output ([ReTool](https://retool-rl.github.io/#:~:text=optimal%20strategies%20for%20leveraging%20external,with%20significantly%20fewer%20training%20steps)) Agent autonomy without structure doesn’t scale. ### **At CodeRabbit, we curate context — we don’t wander** ![Pipeline AI vs agentic AI for code reviews: Let the model reason —](/content/assets/how-we-built-our-ai-code-review-tool-for-ides-inline-image-2.png) We’ve taken a different approach. CodeRabbit’s system: * Runs [30+ static analyzers](https://cloud.google.com/blog/products/ai-machine-learning/how-coderabbit-built-its-ai-code-review-agent-with-google-cloud-run#:~:text=3,20%2B%20linters%20and%20security%20scanners) *before* prompting the model * Uses **AST and symbol lookups** to identify relevant context * Applies context filters based on [past review learnings](https://www.anthropic.com/customers/coderabbit#:~:text=CodeRabbit%20leverages%20Claude%20to%20deliver%3A) * Structures inputs carefully to fit model limits and prompt constraints This hybrid AI pipeline gives the model exactly what it needs — and nothing more. No random guesses, no runtime surprises. We’ve learned that great reviews come from: * Tight, relevant context * Consistent structure * Just enough flexibility to adapt to the code change at hand ### **Could agents learn to curate context?** Maybe — and that’s the interesting future path. If we had: * A dataset of pull requests with “ideal context sets” * Evaluation metrics tied to actual review outcomes * Synthetic examples showing what helps and what hurts... ...then we might be able to train agents to call tools intelligently. To act more like great reviewers than interns with shell access. That’s the direction explored by recent work like [ReTool](https://retool-rl.github.io/#:~:text=optimal%20strategies%20for%20leveraging%20external,with%20significantly%20fewer%20training%20steps) and [LeReT](https://arxiv.org/abs/2410.23214#:~:text=topics,Project%20website%3A%20this%20http%20URL), which use reinforcement learning to teach agents retrieval strategies — learning which tools to invoke and when, based on feedback loops tied to downstream task quality. ReTool showed improvements in task accuracy of up to 9% over retrieval-agnostic baselines, and required significantly fewer training steps to converge. LeReT similarly demonstrated a 29% boost in retrieval success and a 17% gain in downstream QA accuracy over standard retrievers — strong early signals that agents can, in fact, learn to fetch the right context when properly trained. But even with these improvements, we’re still lacking high-quality, domain-specific datasets for tasks like code review. One path forward could involve curating a large-scale benchmark of real and synthetic pull requests, each labeled with: * The issue or defect type present (e.g. logic bug, perf regression, missing test) * The AI context types that improve or degrade LLM performance on detecting that issue (e.g. AST, file diff, related function definitions, ticket description) * The tool invocations used (or simulated) to assemble that context With this dataset, we could: * Evaluate which types of PRs benefit from which types of context * Train agents to learn context selection policies based on PR characteristics * Create specialized sub-agents for different error classes (security, style, performance), each using context proven to enhance detection of those issues In other words: teach agents to reason more like experts — not just by copying what they say, but by emulating how they gather, filter, and apply the information that matters. And we wouldn't have to guess at it. We could back it up with data. That’s the deeper opportunity: not just training agents to run tools, but to understand **why** and **when** to use them — grounded in evidence, driven by outcomes. We’re not there yet. But the path is starting to look clearer. And at CodeRabbit, we’re leading the charge. This is exactly the frontier we’re investing in: building hybrid AI systems that can predict the right tool to use, at the right time, for the right kind of review. Not just to make something clever — but to make something teams can trust. ## **Our hybrid AI pipeline: We reason with purpose** By now, it should be clear that "agentic AI vs. pipeline AI" isn’t the real battle. These are just architectural tools — different shapes we use to tackle the same core problem: *How do we give the model exactly what it needs to deliver a useful review — and nothing that drags it off course?* Pipeline AI systems give us speed, control, and consistency. Agentic AI systems promise adaptability and richer reasoning. And hybrid AI systems, like what we’ve built at CodeRabbit, try to walk that line — combining structure with flexibility, precision with power. But no matter how you structure it, one thing matters above all: **context**. The hard part of code review — for both humans and machines — isn’t the format. It’s knowing where to look. What matters. What can be ignored. What’s risky. What’s surprising. That’s what great engineers learn to spot, and it’s what we’re trying to teach our models to do. That’s the exciting part. Because if we can train a system to not just analyze a diff, but to **know which tool to call, when to call it, and how to interpret its output with surgical precision** — then we’re getting closer to something remarkable. Not just automation. But **reviews that feel like they came from your best engineer** — on their best day — every time. That’s what we’re building toward. Not for the sake of cleverness, but because that’s what teams need: trustworthy tools that help them move fast, write better code, and ship with confidence. We’re not done. But we’re getting closer. ***Interested in trying out CodeRabbit’ reviews?*** [***Get a 14-day trial!***](https://coderabbit.link/yzOslzy)

CodeRabbit for VS Code is here — free AI code reviews

Sahil Mohan Bansal — Wed, 14 May 2025 00:00:00 GMT

We’ve got exciting news to share! CodeRabbit’s AI code reviews are now delivered [directly within VS Code](https://www.coderabbit.ai/ide) and its forks including Cursor and Windsurf. And **code reviews in the IDE are completely free**! With this extension, we focused on building the best quality reviews in the IDE so that we can identify more bugs and issues for you than existing review tools. This brings CodeRabbit directly into your development workflow – making it much easier to incorporate our review feedback in your code. To get started, simply install the CodeRabbit plugin in any VS Code fork. We will then review every commit, identify most bugs before the PR is raised, and help you ship code faster. [![AI code reviews in IDE](/content/assets/ai-code-reviews-vscode-cursor-windsurf-inline-image-2.png)](https://www.coderabbit.ai/ide) *Free AI code reviews in VS Code, Cursor and Windsurf* At CodeRabbit, we believe that the best tools fit seamlessly into your existing workflows. With our AI code reviews coming into Cursor and Windsurf, developers can now get code reviews done without breaking their flow state. ### Benefits of AI code reviews in IDE Code reviews in the IDE are a great way to: * Automate first pass of code review right as you’re coding. * Quickly catch and fix any issues right in your IDE. * Reduce back-and-forth at the PR stage, helping you ship faster. * Help you raise pull requests with greater confidence – and fewer bugs. This multi-layered approach – with reviews in both the IDE and the Git platform – helps developers catch more bugs and reduce review time. Ultimately, code reviews are necessary in both places and deliver complimentary benefits. With reviews in the IDE, developers get a first pass review right as they are coding. With reviews in the Git platform, the entire team’s code is reviewed in a centralized manner, making it easier to catch issues that may creep in when all the commits come together. This helps individual developers get faster feedback in their code editor – while also maintaining a centralized governance structure across the team. [![Code reviews in IDE and in PR](/content/assets/ai-code-reviews-vscode-cursor-windsurf-inline-image.png)](https://www.coderabbit.ai/ide) *Code Reviews in the IDE and in the PR* ### Vibe code with confidence One of the key challenges we hear from developers is how AI coding agents can leave defects behind that are hard to catch. These AI code gen tools are not always able to catch defects in the code they generated. This is why having a second, independent set of eyes is critical to ensure that bugs are caught before they end up in production. Here’s how CodeRabbit works in the IDE: 1. **Inline code reviews:** Each line of code gets senior developer level attention with AI-powered inline review comments. CodeRabbit becomes your pair programmer, within the code editor. [![Line by line code reviews in VS Code](/content/assets/ai-code-reviews-vscode-cursor-windsurf-asset.png)](https://www.coderabbit.ai/ide) 2. **Built for flow, not friction:** Code, review, commit - without breaking your flow state. CodeRabbit reviews every committed and uncommitted change, making your dev workflow faster. [![review committed and uncommitted changes](/content/assets/ai-code-reviews-vscode-cursor-windsurf-asset-2.png)](https://www.coderabbit.ai/ide) 3. **Fix-with AI:** Some review suggestions are simple and can be incorporated by using “One-Click Fix” to apply changes instantly. For more complex feedback, our “Fix with AI” feature hands off CodeRabbit’s review comments with associated context to your preferred AI coding agent. [![Fix with AI](/content/assets/ai-code-reviews-vscode-cursor-windsurf-asset-3.png)](https://www.coderabbit.ai/ide) 4. **Fork-compatible and language agnostic:** Our VS Code plugin supports all VS Code forks including Cursor and Windsurf. Also, all commonly used languages are supported such as Java, JavaScript, PHP, Python, TypeScript, Go, Ruby and many more. ### IDE reviews are a lightweight, faster version of our PR reviews Our free reviews in the IDE are done by a lightweight version of CodeRabbit that delivers feedback instantly and catches most bugs. For production use-cases, we recommend using the full version of CodeRabbit that also reviews code in the PR, in addition to reviews in the IDE, and comes with additional features that includes additional context from several sources during the review process. Check out the technical details of how we adapted our PR reviewer for [instant reviews in the IDE](https://www.coderabbit.ai/blog/how-we-built-our-ai-code-review-tool-for-ides) Note that the rate limits for reviews in IDE will be lower than rate limits for reviews in PR. For a detailed feature comparison of what is included with reviews in IDE vs reviews in PR, [refer to our documentation](https://docs.coderabbit.ai/code-editors) ### Next Steps Try out CodeRabbit’s free AI code reviews in VS Code, Cursor or Windsurf, we’d love to hear your feedback. Setup takes just a few minutes. Here are some additional resources to help get started: * [Install the VS code plugin](https://marketplace.visualstudio.com/items?itemName=CodeRabbit.coderabbit-vscode) * [Watch demo video](https://coderabbit.link/vscode-demo) * [Share feedback on Discord](https://discord.gg/coderabbit) * [Contact Sales for higher rate limits](https://www.coderabbit.ai/contact-us/sales)

How we built our AI code review agent for IDEs

Amitosh Swain — Wed, 14 May 2025 00:00:00 GMT

At CodeRabbit, we recently shipped our [free VS Code extension](https://coderabbit.link/rxv5kZ8), bringing context-rich AI-powered code reviews directly into your editor. Our engineering philosophy has always been simple: we build tools that fit seamlessly into your existing workflow. While developers have told us our [comprehensive PR reviews](https://docs.coderabbit.ai/) have helped them ship faster and keep more bugs out of production, many also asked for [IDE reviews](https://coderabbit.link/ozj7K45) to help check code before sending a pull request. By creating another review stage within VS Code (and compatible editors like Cursor and Windsurf), we've minimized disruptive context switching, allowing developers to catch logical errors and embarrassing typos before they ever send a PR. In our dogfooding, our team has found it helps catch issues early and reduces the iterative back-and-forth that can slow down teams at the pull request stage. One engineering challenge we faced when designing IDE reviews was re-engineering our code review pipeline to meet developers’ expectations for instant reviews in their IDE. In this post, we’ll share how we thought through this shift and what steps we took to ensure high-quality reviews while reducing the time-to-first comment by ~90%. ## Transforming CodeRabbit’s typical code review pipeline ![Architectural diagram for CodeRabbit's PR reviews](/content/assets/how-we-built-our-ai-code-review-tool-for-ides-inline-image.png) Because CodeRabbit initiates a review immediately after a PR is sent, we optimized our PR code review process for the most helpful reviews and send a notification to the reviewer when our review is completed. That process takes several minutes and involves a complex, non-linear pipeline that pulls in dozens of contextual data points for the most codebase-aware reviews. ![How we built our AI code review agent for IDEs](/content/assets/how-we-built-our-ai-code-review-tool-for-ides-inline-image-2.png) We then subject all recommendations to a multi-step verification process to validate that each suggestion will actually be helpful in order to ensure our reviews have as little noise as possible. This involves multiple passes where we identify how parts of the codebase and changes fit together to look for issues before also processing and verifying each suggestion separately. During this process, we don’t share any of those suggestions with the user but wait until we’ve completed all steps in the pipeline in case we find that a suggestion might not be useful. This approach creates a sophisticated, non-linear review graph that is delivered in a batch to minimize noise. However, users receiving a PR are unlikely to act on it immediately, so the processing time isn’t noticed. In that case, it makes sense to bias for quality over speed. But it’s different with IDE reviews. In the IDE, the user expects that reviews will start instantly and be delivered quickly since they might be waiting for the review before sending a merge request. They want to start working on changes immediately. To create an IDE review that better fit user expectations, we needed to adjust how we approached reviews for that context—without impacting review quality or usefulness. ## Balancing quality and speed ![How we built our AI code review agent for IDEs](/content/assets/how-we-built-our-ai-code-review-tool-for-ides-asset.png) Developing a streamlined pipeline able to deliver near real-time feedback directly within your coding environment required that we restructure how events and review comments were both created and transmitted. This process started by thinking intentionally about the affordances around creating valuable insights in real time. Real-time reviews would make it impossible for every comment to go through our verification steps to ensure its relevance. But removing those steps from our pipeline would lead to noisier reviews and a higher proportion of unhelpful suggestions. At what point would the signal-to-noise ratio make an AI code review more of a nuisance than a help? Since we also offered a solution at the PR stage, we could make our IDE reviews narrowly focused on what mattered most at that stage. We decided that anything that was architectural or required deeper verification with the full codebase was better suited for the PR stage since trying to deliver valuable insights around that would require multi-step validation and take too long. In our PR reviews, we even go so far as to check code by attempting to run it in our sandbox. But that would be impossible in real-time. We decided instead to prioritize reviewing for mistakes, specification alignment, bugs, and logical issues that a developer might miss while coding or editing. Since those are the types of things that could require a PR to be sent back for changes or make your colleagues question *how you could have missed them*, we saw these as the most critical issues to look for in the IDE. For example, in my own use of our IDE reviewer, it found a conditional that I changed by mistake and didn’t notice. We also still wanted to have a verification process that would validate that any suggestions we made would be beneficial but needed to build a more lightweight process for doing so. Our goal was to develop the best IDE reviews, but we knew they didn’t need to be as comprehensive as our PR reviews. The goal of IDE reviews is to streamline the PR process with an IDE check to help devs tackle the most critical changes and merge more confidently. At that point, the code would undergo our more in-depth PR review. ## How we engineered it ![How we built our AI code review agent for IDEs](/content/assets/ai-code-reviews-vscode-cursor-windsurf-asset.png) Here’s how we tackled developing a new pipeline for our IDE reviews. ### Review processing and delivery One of the biggest changes we made was to how we process and deliver reviews. While in the SCM, we process all suggestions together and wait until we’ve completed our review before delivering the results to users, in the IDE we opted to deliver suggestions iteratively in real-time as our pipeline created them. While we could have opted for a more superficial review and delivered the complete results faster, we wanted to balance comprehensiveness and quality with the user’s need for instant feedback. By sharing suggestions over a longer period of elapsed time, we were able to buy extra time to include additional steps in the process to improve the quality of the suggestions. Because we opted to engineer our system in this way, that also meant that CodeRabbit was able to give continuous feedback as you code using the same review process and pipeline. ### Context preparation ![How we built our AI code review agent for IDEs](/content/assets/ai-code-reviews-vscode-cursor-windsurf-asset-2.png) Context enrichment is a big part of how we deliver the most codebase-aware and relevant recommendations. However, our context preparation process is extremely comprehensive for our PR reviews. We go so far as bringing in linked and [past issues](https://docs.coderabbit.ai/guides/linked-issues), cloning the repository, and even building a [Code Graph](https://docs.coderabbit.ai/integrations/code-graph-analysis/) of your codebase to analyze dependencies. That kind of context enrichment wasn’t possible in the IDE. Instead, we focused on the code primarily to keep the context lighter for a faster review. ### Prompt and model optimization Despite rearchitecting our pipeline to be more linear and to cut down on steps, we didn’t alter the multi-model orchestration that we use for our PR reviews. We use the same orchestration of models for both kinds of reviews, but we created different choices of weights to intelligently select the models to process different parts of the review. We also fine-tuned how our decisioning engine worked to create a more linear process flow. Finally, we optimized our prompts for faster response times and the different priorities we’d identified for IDE reviews. ### Choosing non-streaming LLM responses over streaming responses We first thought streaming responses—generating words one by one from a language model like in ChatGPT—would be ideal for our IDE reviews, especially since it’s popular for real-time tasks. But because our prompts are large and we perform significant context engineering before starting our reviews, we ran into problems like garbled output from the model and missing tool calls. Users expect review comments to be complete, unlike casual chats with AI coding assistants. So, we have to clean up the model’s output before showing it. Since streaming models generate their output in chunks, we had to buffer the LLM output until the model generated a full comment before sending it through our processing pipeline. This delay meant streaming didn’t help much in our case. Instead, we chose to wait a bit longer to get complete outputs for a bunch of files simultaneously. ### UI changes ![How we built our AI code review agent for IDEs](/content/assets/ai-code-reviews-vscode-cursor-windsurf-inline-image-2.png) We had to design our UI from the ground up in VS Code. At first, we thought of adding our own comments panel where we would add comments similar to how we do it in SCMs. But we realized that a comments panel wasn’t the right way to do real-time comments. We decided to integrate our comments more directly into the editor so that users get the comment where the code is. We find this is a more IDE-native strategy and creates a better UX flow. ### Working with users’ preferred AI coding agents Because so many developers use AI coding tools in their IDE, we wanted to take advantage of that to give devs a choice in how to resolve a suggestion from CodeRabbit. We give users the option to resolve the issue with code we suggest or to pass the suggestion to the user’s preferred AI coding assistant to suggest code, all in one click. ## What’s next on the roadmap? ![How we built our AI code review agent for IDEs](/content/assets/how-we-built-our-ai-code-review-tool-for-ides-asset-2.png) We might have launched version 1.0 of our IDE reviews, but we have a number of things on our roadmap to make them even more helpful. ### Near term * **User-level Learnings:** We’ll be adding the ability to add Learnings or give feedback on suggestions so our agent automatically learns the suggestions you like and don’t like. We currently have org-wide Learnings in the SCM but want to extend this feature to individual developers who want to add custom Learnings that will only apply to them. %[https://www.youtube.com/watch?v=Yu0cmmOYA-U] * **Tools:** We’re planning on bringing the [30+ tools](https://docs.coderabbit.ai/tools/) you can use in our PR reviews to our IDE reviews. If you’re already running linters in your IDE, you’ll be able to do it all in one review with CodeRabbit. Look out for this addition later in the year. ### Longer-term * **Web Queries:** We plan to integrate our context-enhancing features into our IDE tool to help identify and prevent bugs with fewer false positives and keep reviews up-to-date on versions, library documentation, and vulnerabilities, even if the LLM isn’t. * **Docstrings:** Want to create [docstrings](https://docs.coderabbit.ai/finishing-touches/docstrings) before you merge? We’ll be adding this feature that’s currently part of our PR reviews to our IDE reviews in the future. * **More codebase awareness:** We will be bringing in additional data points to create more codebase-aware reviews in the IDE. ***Try our free*** [***IDE reviews here***](https://coderabbit.link/rxv5kZ8)***. Interested in tackling similar challenges?*** [***Join our team.***](https://coderabbit.link/ecTThbN)

CodeRabbit now supports AI code reviews with Bitbucket Cloud

Sahil Mohan Bansal — Tue, 04 Feb 2025 00:00:00 GMT

We are thrilled to announce that CodeRabbit [now supports](https://docs.coderabbit.ai/platforms/bitbucket-cloud/) AI Code Reviews with Bitbucket Cloud! This integration, currently in beta, delivers a wider choice of supported Git platforms including GitHub, GitLab, Azure DevOps, and Bitbucket. CodeRabbit’s AI Code Reviews help customers automate the first pass of code reviews, reduce manual time spent in the review cycle, catch bugs and issues that can be overlooked due to human error, and improve the overall [quality](https://www.coderabbit.ai/blog/code-reviews-made-easy-how-to-improve-code-quality) of their code. With support for Bitbucket Cloud, we are making it even easier for developers to automate away manual and tedious tasks, thus freeing up their time to focus on what they care about: developing new features. ### **Benefits of using CodeRabbit:** * **Automated code reviews:** Automatically reviews your code and identifies potential issues, such as bugs, security vulnerabilities, and refactoring opportunities. * **Best-in-class reviews:** Logical reasoning capabilities provided by Gen-AI models integrated with linters deliver a high signal-to-noise ratio for best-in-class AI reviews. * **Configurable Learnings:** Your code reviews are customizable to your specific coding standards. AI automatically learns over time and improves future reviews. * **Docstrings:** Save time with unit-test insertion and docstring generation. Focus developer time on more productive activities while automating tedious tasks. * **Just minutes to get started:** Just give access to your Bitbucket, create a new PR, and AI code reviews will kick in automatically. Customers who have implemented AI code reviews have reported: * 2x-4x faster PR merge time * 25% or more reduction in the amount of dev time spent on code reviews * reduction in the number of bugs in production in the first week after implementing CodeRabbit *“What sets CodeRabbit apart is its deep understanding of code structure through AST analysis. It's not just pattern matching—it's intelligent code comprehension that integrates seamlessly into our existing workflows” – Ron Efroni, Founder, FloxDev* Check out more [customer testimonials](https://www.coderabbit.ai/customers). ### **How to get started with CodeRabbit and Bitbucket:** 1. Start a free trial or log in to your existing account. Select Bitbucket as your Git provider at login. 2. Create a new Bitbucket user as a service account and name it “CodeRabbit.” Then, associate this user with the workspace to which you would like to give CodeRabbit access. 1. Generate an App Password to enable integration between CodeRabbit and your Bitbucket repositories. Enter the App Password in the CodeRabbit UI. This will automatically configure the required webhook for seamless integration. ![CodeRabbit now supports AI Code Reviews with Bitbucket Cloud](/content/assets/coderabbit-now-supports-ai-code-reviews-with-bitbucket-cloud-asset.png) *Login with Bitbucket as your Git provider* ![CodeRabbit now supports AI Code Reviews with Bitbucket Cloud](/content/assets/coderabbit-now-supports-ai-code-reviews-with-bitbucket-cloud-asset-2.png) *Enter your App Password in the CodeRabbit UI* You can read more [detailed documentation](https://docs.coderabbit.ai/platforms/bitbucket-cloud/) on the Bitbucket integration and join us on our [Discord server](https://discord.gg/coderabbit) for real-time support. Please note that Bitbucket support is currently in beta, so there may be a few kinks that will be resolved over time. Please report any issues on the “[Bitbucket Cloud Support](https://discord.com/channels/1134356397673414807/1334660150623076473)” Discord channel. Start your [14-day free trial](https://app.coderabbit.ai/login?free-trial=) today and implement AI Code Reviews to immediately improve your code quality, reduce bugs in production, and free up developer time.

CodeRabbit now available on GCP Marketplace

Manpreet Kaur — Mon, 23 Dec 2024 00:00:00 GMT

Following the recent announcement of CodeRabbit being available on AWS Marketplace, the CodeRabbit team is excited to announce that we are now also available on [GCP Marketplace](https://console.cloud.google.com/marketplace/product/coderabbit-public/coderabbit)! Enterprise customers aiming to adopt AI code reviews to deliver higher-quality code more efficiently can now utilize their current spending agreements with GCP to finance CodeRabbit. This enables users to quickly embrace AI for their code reviews without needing to endure extensive budgetary approvals that can often take months, thus delaying the adoption of AI in their software development workflow. Customers can utilize their spending commitments with GCP to purchase CodeRabbit in only a few clicks, allowing the implementation of AI code reviews in days, not months. Here are the key details: * [CodeRabbit Enterprise](https://www.coderabbit.ai/enterprise) plan is supported on GCP Marketplace. * Customers get CodeRabbit’s self-hosted container image that they can host in their own GCP environments. * The total purchase amount via GCP Marketplace is eligible to be counted toward customers’ spend commitments with GCP. * The minimum developer seat count for a GCP Marketplace purchase is 500 seats. * Support available 24/7 and dedicated engineering onboarding assistance are included with every purchase from the Marketplace. Contact the CodeRabbit sales team ([sales@coderabbit.ai](mailto:sales@coderabbit.ai)) for discounted pricing for bulk purchases in the form of a GCP Marketplace private offer that provides better rates tailored to your requirements. This represents a major advancement in assisting enterprise clients with the difficulties posed by the growing volume of code produced through AI code generators, even as the code review process continues to be predominantly manual. With the rapid adoption of tools such as GitHub Copilot, Cursor, and others, the code generation process has been significantly augmented with AI. Companies aiming to quickly deploy features at scale must also enhance their manual code review process by incorporating automated AI code reviews. Here is how CodeRabbit can help such customers: 1. **Merge PRs 2x–4x faster:** AI streamlines the initial phase of your code review process and detects bugs or errors in the code considerably faster than manual assessments, enabling you to merge your pull requests (PRs) 2x to 4x quicker and deliver more features with the same team strength. 2. **Fix over 90% of bugs and errors:** Ensure no crucial mistakes get to production. Manual evaluations may be susceptible to mistakes and occasionally overlook a significant error, which can be quite costly to rectify in production. By utilizing AI code reviews, you will consistently remain aware of crucial errors and address them during the code review process before they affect any customers. 3. **Enhance code quality:** AI code evaluations get better with consistent usage. This elevates your development team’s capabilities by ensuring an expert in all languages is continuously accessible for you directly within your [Git platform](https://www.coderabbit.ai/blog/coderabbit-now-supports-ai-code-reviews-with-bitbucket-cloud). 4. **Maintain data privacy:** Store your code reviews and evaluations within your own infrastructure. Execute CodeRabbit’s container image in your self-hosted GCP container to maintain full control over data privacy and guarantee alignment with your security policies. Thousands of customers have embraced CodeRabbit to enhance their code review workflows with AI assistance. You can see their statements on [our website](https://www.coderabbit.ai/). Ready to implement AI Code Reviews in your software development lifecycle? Reach out to [sales@coderabbit.ai](mailto:sales@coderabbit.ai) and we would love to get you going with CodeRabbit on GCP.

CodeRabbit now available on AWS Marketplace

Sahil Mohan Bansal — Tue, 17 Dec 2024 00:00:00 GMT

The CodeRabbit team is excited to announce that we are now available on [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-wkkkre4fgelwq)! Enterprise customers looking to implement AI code reviews in order to ship higher-quality code in less time can now leverage their existing spend commitments with AWS to pay for CodeRabbit. This allows customers to more rapidly adopt AI to power their code reviews without having to go through lengthy budgetary approvals. Customers can use their spend commitments with AWS to pay for CodeRabbit in just a few clicks. Here are the key details: * [CodeRabbit Enterprise](https://www.coderabbit.ai/enterprise) plan is supported on AWS Marketplace. * Customers get CodeRabbit’s self-hosted container image that they can host in their own AWS EKS or ECS environments. * The total purchase amount via AWS Marketplace is eligible to be counted toward customers’ spend commitments with AWS. * The minimum developer seat count for an AWS Marketplace purchase is 500 seats. * 24/7 support and dedicated onboarding engineering support are included with all Marketplace purchases. Reach out to the CodeRabbit sales team ([sales@coderabbit.ai](mailto:sales@coderabbit.ai)) for discounted pricing for bulk purchases in the form of an AWS Marketplace private offer with more attractive pricing that meets your needs. This is a significant step forward in helping enterprise customers deal with the challenges of the ever-increasing quantity of code being generated using the power of AI, while the code review process remains largely manual. With the rapid adoption of tools such as GitHub Copilot, the code generation process has been augmented with AI. Enterprises who want to ship features rapidly, at scale, need to also augment their manual code review process with automated AI code reviews. Here is how CodeRabbit can help such customers: 1. **Merge PRs 2x–4x faster:** AI automates the first pass of your code review cycle and finds any bugs or errors in the code much faster than manual reviews, allowing you to merge your pull requests (PRs) 2x to 4x faster and ship more features with the same team size. 2. **Catch 90%+ of bugs and errors:** Prevent a critical error from reaching production. Manual reviews can be error-prone and sometimes miss a critical error that can be very expensive to fix in production. With the power of AI code reviews, you will always stay on top of any critical errors and catch them during the code review cycle before they impact any customers. 3. **Improve code quality:** AI code reviews improve over time the more you use them. This raises the floor of your development team by having an expert in all languages always available that you can leverage directly in your existing Git platform. 4. **Maintain data privacy:** Keep your code reviews and analysis in your own private infrastructure. Run CodeRabbit’s container image in your self-hosted environment for complete control over data privacy and ensure compliance with your security policies. Thousands of customers have already adopted CodeRabbit to improve their code review process using AI. You can read what they are saying on [our website](https://www.coderabbit.ai/). Ready to implement AI Code Reviews in your software development lifecycle? Reach out to [sales@coderabbit.ai](mailto:sales@coderabbit.ai) and we would love to get you going with AI Code Reviews. *P.S.—Yes, support for GCP and Azure Marketplaces is coming very soon too!*

Announcing the CodeRabbit Startup Program

Aravind Putrevu — Mon, 30 Sep 2024 00:00:00 GMT

In the startup world, speed and quality are often seen as tradeoffs; with AI-powered code reviews, you can have both. Startups and small dev teams face numerous challenges shipping code day-to-day while navigating tight deadlines and meeting user expectations with limited resources. Every phase could be a leap to the next level of growth, thereby improving the chance of standing out. At CodeRabbit, we understand the importance of shipping better-quality code. Being a startup ourselves, we understand how much it matters to ship bug-free code, thereby enabling better customer experiences. We are thrilled to announce our new Startup Program, designed to bring the power of AI-powered Code Reviews to emerging tech companies. ## CodeRabbit Startup Program Our startup program is designed to give early-stage companies access to AI-powered code review technology. We understand that when you are building something new, every resource counts. CodeRabbit helps you to maximize your development team's potential. Our innovative system enhances [**code quality**](https://www.coderabbit.ai/blog/code-reviews-made-easy-how-to-improve-code-quality) through intelligent PR reviews, ensuring your codebase remains robust and maintainable. We've also integrated advanced [**code security**](https://www.coderabbit.ai/blog/security-at-ai-speed) measures, safeguarding your applications against vulnerabilities. Our AI **copilot for pull requests** streamlines the review process, catching issues early and facilitating smoother collaborations. All of these features work in concert to boost overall **developer productivity**, allowing your team to focus on what truly matters—building and shipping great products. ### What's included in the Startup Program? * 50% off for 6 months on a CodeRabbit **Pro Plus annual plan** after the standard 14-day free trial period. * Support * Access to the CodeRabbit community on Discord [Discover the features and benefits of the Pro Plus Plan](https://www.coderabbit.ai/pricing) ### Eligibility Criteria * VC or accelerator-backed startups that do not yet have a CodeRabbit account * Funding of less than $25 million ## Benefits of using AI-powered Code Reviews CodeRabbit is your invisible Senior Engineer, reviewing every code change in real time. We could help you improve by: * Accelerating your development cycles * Improving code quality and security coverage—PR by PR. * Retaining best-practice learnings from human reviewers at an organization level. * Improving developer and reviewer productivity by cutting bugs and review time in half. ## Getting Started with the Startup Program * Fill out the [Startup Program application form](https://www.coderabbit.ai/startup-program). * Our team will review your application within 2 business days. * You will receive an email with details of your approval. At CodeRabbit, we believe that AI is not here to replace developers but to improve their productivity and empower them. By handling the repetitive aspects of code review, [CodeRabbit](https://www.coderabbit.ai/) frees up your team to focus on what they do best: creating the best business outcomes! We are excited to partner with the next generation of tech companies and help them build robust, scalable solutions that will shape our future in many ways. Join our [startup program today](https://www.coderabbit.ai/startup-program) and experience AI-driven code reviews.

CodeRabbit announces $16M Series-A funding led by CRV

Aravind Putrevu — Tue, 13 Aug 2024 00:00:00 GMT

Exciting news! CodeRabbit has secured a [$16 million Series A funding round](https://www.crv.com/companies/coderabbit), with [CRV](https://www.crv.com/) leading the charge. This funding will help us accelerate our mission to transform code quality, security and developer productivity with AI. [Reid Christian](https://www.crv.com/team/reid-christian), General Partner at CRV, will be joining our board, and we're thrilled to leverage CRV's expertise and network to serve our customers better. For us, finding the right early-stage VC firm was all about increasing our productivity and accelerating our growth. Selecting CRV as our partner was easy given the firm's commitment to the developer community and its track record for working alongside founders, not just in the startup phase, but through IPO and beyond. This round includes investments from [Tod Sacerdoti (Flex Capital)](https://www.flexcapital.com/), [Ashmeet Sidana (Engineering Capital)](https://www.engineeringcapital.com/) and several mentors, who bring a wealth of experience in developer tools and enterprise AI. ## Our Journey So Far We are witnessing an inflection point in the software development industry. Developers around the world have been realizing the incredible possibilities that AI can bring. The introduction of GitHub Copilot and ChatGPT have revolutionized software development and have been the fastest-growing tools in this space. While on the code generation side, many tools have emerged, the code review process has remained largely unchanged. We continue to use the same tools and processes that were used 10 years ago. The code is still manually reviewed, which is slow, error-prone, and expensive. CodeRabbit was born out of this realization. As developers ourselves, we faced the inefficiencies and frustrations of traditional code reviews at our previous workplaces. We envisioned an AI-powered code reviewer that could save developers countless hours while improving the quality of the code that goes beyond what is possible with human reviewers alone and existing linting tools. Since its inception, CodeRabbit has been well-received by the developer community. The company reached $1M+ in annual recurring revenue milestone in less than a year, while operating on a bootstrapped budget. Some high-level traction stats: * Most installed AI application on GitHub Marketplace: 16K+ installations * 150K+ repositories under review * Several hundred paying organizations * High adoption in open-source projects: 300K+ pull requests reviewed * On average, 10K+ developers use CodeRabbit daily * Strong community: 15K+ followers on X and 2K+ users on our Discord server ## What Worked for Us * **Intuitive product experience**: Investing in sheer delight and attention to detail helped build a loyal user base. This is especially true in the AI-applications space, where user experience can make or break a product. Most AI products tend to be spammy and noisy, leading to user abandonment. Right from the get-go, we strived to maintain a high signal-to-noise ratio in our product and provide feedback that is actionable and easy to accept without cognitive overload. * **Engineered for product-led growth:** Early on, we realized that the best way to bring our product to a wider audience was to make it easy for developers to discover and use it. We invested heavily in the seamless onboarding experience, easy organization-wide rollouts, integration with popular platforms like GitHub & GitLab, easy to understand pricing, and a generous free tier. This helped us grow our user base rapidly and organically. * **Getting love with the open-source community**: We made our paid product free for public repositories, which helped us gain a strong foothold in the open-source community. This not only helped us build a strong brand presence but also allowed us to get valuable feedback from the community, which we used to improve our product. We strongly believe in giving back to the community and have been an active sponsor of several open-source projects that we use in our stack. ## What Lies Ahead CodeRabbit is a fast-growing company, and we are excited about the future of bringing AI to developer workflows. The new venture funding will help us hire top talent across engineering, product and GTM functions to help us expand our product offerings and accelerate our growth. Ready to join us? Explore open positions on our [careers page](https://www.coderabbit.ai/careers) or email us at [careers@coderabbit.ai](mailto:careers@coderabbit.ai). Not looking for a new role but want to supercharge your code reviews with AI-driven contextual feedback? Try CodeRabbit with a [free 14-day trial](https://www.coderabbit.ai/)!

Code reviews made easy: How to improve code quality

Aravind Putrevu — Tue, 16 Apr 2024 00:00:00 GMT

The fastest way to tank a team's ability to improve code quality is to exhaust the people responsible for maintaining it. Most teams know how to improve code quality in theory: write tests, enforce standards, review every pull request (PR). The hard part is doing all of that without grinding senior engineers into dust. Adding more checklists, more review gates, and more meetings on top of an already-stretched team leads to resignations, not better software. This piece covers the practices, metrics, and workflow patterns that actually improve code quality while keeping cognitive load manageable. You’ll learn how to automate the cheap stuff, [shrink the review surface](https://www.coderabbit.ai/blog/introducing-change-stack-the-first-ai-native-code-review-interface), and save human judgment for the decisions only humans can make. ## What "code quality" actually means in modern development Code quality in modern development is the degree to which an engineer unfamiliar with the codebase can change it correctly. That breaks down into a few concrete properties: * **Maintainability:** A non-trivial change doesn't break something the author didn't know existed. * **Testability:** There's a reliable way to know the change worked before it ships. * **Security:** Credential handling, input validation, and the failure modes that turn into post-mortems. Code quality issues compound as [technical debt](https://www.coderabbit.ai/blog/vibe-coding-because-who-doesnt-love-surprise-technical-debt). The work doesn't stop; it just costs more every time. The [Stack Overflow Developer Survey 2024](https://survey.stackoverflow.co/2024/professional-developers) confirmed that 63% of professional developers cite technical debt as their number-one frustration at work. Quality debt slows your team even more than it slows the code, and the engineers who feel it first are the senior ones doing the most reviews. ## Why traditional code quality practices break at scale [Code review bottlenecks](https://www.coderabbit.ai/blog/nobody-is-going-to-read-the-code) come more from cognitive load than scheduling. As teams grow, the volume of work that needs reviewing grows faster than the pool of senior engineers qualified to review it. Three structural patterns explain how that gap turns into a break: * **PR size:** Reviewers skim long diffs. Once a diff crosses the threshold of what one person can hold in their head, careful review turns into pattern-matching for the obvious stuff. * **Reviewer load:** Senior engineers carry the worst of the review burden, on top of architectural review, security review, and mentorship. [JetBrains' Developer Ecosystem Survey 2025](https://blog.jetbrains.com/research/2025/10/state-of-developer-ecosystem-2025/) confirms that coordination responsibilities increase with experience. * **AI-generated volume:** AI tools ship more PRs, and each PR carries more issues than human-only code. [CodeRabbit's State of AI vs Human Code Generation Report](https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report) analyzed 470 open-source PRs and found AI-co-authored PRs produce 10.83 issues per PR on average, compared to 6.45 for human-only PRs. As a result, reviewer fatigue compounds, bugs slip through, and coding conventions fade because nobody has time to enforce them. Eventually, a senior engineer leaves, and the team realizes nobody else knew what the conventions were. ## How to assess code quality (and know it's improving) Two metrics do the load-bearing work: cycle time and defect escape rate. Pick a baseline, watch the trend, and the rest sorts itself out. **Cycle time** is the time from when a PR opens to when it merges. It captures most of what goes wrong in review. If it's climbing week over week, something has shifted. PRs are getting bigger, reviewers are slower, or comments are stalling on low-priority issues instead of substance. Cycle time hides two distinct problems, and the diagnosis matters. Long time-to-first-review means reviewer capacity is the constraint, so the fix sits on the reviewing side: automate the first pass, distribute load, cap incoming PR size. Long time-in-review-cycles means feedback is unclear or comments stall on low-stakes issues, so the fix sits on the comment side: clearer review standards, written norms, prioritized blocking comments. Track cycle time at the team level, not the individual level. Compare against your own historical baseline rather than an industry number. **Defect escape rate** is the second metric worth watching. What percentage of bugs reach production instead of getting caught in review? Down means quality is improving. Up means something is broken upstream. If you don't tag bugs as "should have been caught" versus "genuinely subtle," start. The category split is where the actionable signal lives. Numbers tell you what happened, but qualitative signals tell you what's about to happen. Watch for: * **Reviewer complaints:** Senior engineers volunteering frustration about review load is a leading indicator before any metric moves. * **Author complaints:** Junior engineers saying their PRs sit too long is the same signal from the other end. * **Review skipping:** PRs getting merged without review during crunch weeks means quality is already breaking down upstream of anything you'd track. You know it's working when cycle time trends down or stays flat as PR volume grows, defect escape rate drops, first-review waits shrink, and senior engineers stop complaining about review burden. None of these are numbers you can hit overnight. They move on quarter-scale, not week-scale, and the most reliable way to read them is against your own history. ## Best practices to improve code quality without killing velocity Five practices move the needle on code quality the most, and none of them are new. The reason teams still ship buggy code is not that the practices are unknown; it's that the cost of executing them consistently has been higher than the cost of skipping them. ### 1. Keep PRs small Small PRs are one of the biggest levers on review quality. Long diffs invite skimming, and skimming is how bugs reach production. A change small enough to hold in your head lets a reviewer give substantive feedback instead of pattern-matching across a sprawling diff. [Google's engineering practices documentation](https://google.github.io/eng-practices/review/developer/small-cls.html) backs this up and gives reviewers explicit authority to reject changes solely for being too large. Smaller review units also reduce reviewer fatigue and move feedback through faster. ### 2. Adopt trunk-based development Trunk-based development is the practical enforcement layer for keeping PRs small. When everyone integrates into a shared trunk within a day or two, big speculative changes never get a chance to accumulate. Merge conflicts shrink, and reviewers see incremental work instead of multi-week dumps. Feature flags handle the parts that aren't ready to ship, so the team can keep merging without exposing half-built features to users. The combination gives you frequent, low-risk merges and keeps the team in the habit of reviewing fresh code instead of archaeology. ### 3. Automate testing Automated testing moves quality enforcement to the cheapest place to fix problems: before the code reaches a reviewer. A reliable suite of unit tests for logic, integration tests for boundary behavior, and a small set of end-to-end tests for critical paths catches the failures that would otherwise eat review time. Tests run in continuous integration (CI) on every PR, fail loudly, and block the merge until resolved. The point isn't 100% coverage; it's coverage on the paths that actually fail. Done right, human reviewers stop verifying basic correctness and start focusing on design, edge cases, and the failure modes the tests don't cover. ### 4. Layer static analysis and linting Static analysis and linting belong in the pipeline as automated guardrails, not human checklists. Run them pre-commit or in CI so structural issues, type mismatches, common security patterns, and style violations get flagged before any reviewer sees the diff. The mechanical layer should catch the mechanical problems. A reviewer who has to comment on missing null checks or unused imports is spending cognitive load that should go toward design and logic. Combine the tools rather than picking one: * A linter for style * A type checker for correctness * A static application security testing (SAST) scanner Each catches a different class of issue, and the overlap is small. ### 5\. Standardize coding conventions Shared coding conventions remove the most predictable arguments from code review. Document the patterns your team agrees on: naming, file organization, error handling, testing structure, and what "done" actually means for a PR. Make as much of it machine-enforceable as you can. A formatter handles whitespace, while a PR template enforces the definition of done. The work of standardization happens once, and it pays out every time a reviewer doesn't have to explain a convention from scratch. When conventions live only in a senior engineer's head, drift is inevitable. But when they live in code and tooling, drift is hard. ![Freee logo with a bird icon and 'CodeRabbit CASE STUDY' text on a dark grid background.](/content/assets/free.png) Together, these practices compound. [For example, freee](https://www.coderabbit.ai/case-studies/how-freee-saved-months-of-reviewer-time-coderabbit), a Tokyo-based business management SaaS company, saved the equivalent of 32.8 weeks of reviewer time over six months after pairing these practices with AI code review. The deployment expanded from a 30-seat pilot to 570 seats across 285 repositories. ## A practical code review checklist When the team agrees on what every reviewer should check, standards drift disappears. A useful checklist covers six things: * **Functionality:** Does the code do what it claims? Are edge cases handled? Are failure paths tested? * **Logic and correctness:** Conditional logic, error handling, null safety, and concurrency. These are the categories [CodeRabbit's State of AI vs Human report](https://www.coderabbit.ai/blog/state-of-ai-vs-human-code-generation-report) flags most heavily in AI-generated code. * **Readability:** Variable names, structure, and comments where complexity warrants them. Skip the formatting wars. Let the formatter handle those. * **Security:** Credential handling, input validation, and common injection patterns. Catch these in review, not in a post-mortem. * **Tests:** Does the test cover the behavior, not just the implementation? Do the tests fail when the code is wrong? * **Documentation:** Public APIs, non-obvious decisions, and breaking changes need a note. That's the human reviewer's job. Style, formatting, and obvious linting violations should be caught earlier in the pipeline. ## Designing code review for async, distributed teams The cost of async review is round count. Each back-and-forth eats a day, so the teams that ship across time zones optimize to land the review in one or two passes, not to be faster, but to require fewer rounds. Four practices keep the round count low: * **Pre-annotate non-obvious code in the PR description:** Walk the reviewer through the changes before they see the diff. The first round of "what is this doing?" should be answered before they ask. * **Make every comment self-contained:** Include the actual question and the actual ask. "Thoughts?" forces a follow-up. "Can we extract this into a helper to avoid duplicating the validation logic?" gets answered in one round. * **Mark comments as blocking vs. non-blocking:** Authors need to know which comments must be addressed before merge and which are suggestions. Without that signal, every round becomes a clarification round. * **Set a primary reviewer:** Multiple reviewers in different time zones, each waiting on the others, means PRs rot. Name one primary; mark the rest optional. The teams that handle this well treat async review as a design problem, not a scheduling problem. ![Taskrabbit logo with 'CodeRabbit CASE STUDY' text on a dark, textured background.](/content/assets/taskrabbit.png) [Taskrabbit](https://www.coderabbit.ai/case-studies/taskrabbit-cut-merge-time-by-25-before-adopting-coding-agents) fixed exactly this before adopting AI coding agents. With engineering split between San Francisco and Poland, PRs opened after 2 pm PST waited a full day for the first review. After deploying CodeRabbit, the average PR cycle time dropped from 10 days to 7, a 25% reduction. Senior Engineering Manager Kiran Kanagasekar described the logic plainly: "Writing code faster was never the issue; the bottleneck was always code review." ## Putting it all together: AI code review and the sustainable quality workflow AI code review works as a first-pass filter that preserves human reviewer capacity for the decisions that actually need it. The categories AI handles well are the ones that wear humans out: catching null checks the author forgot, flagging unused imports, surfacing common security patterns, and spotting style inconsistencies. The categories where humans still win are the ones that require domain context: design, architecture, and the question of whether this change makes sense for what the team is actually trying to build. The split is clean: * **AI handles volume:** Null checks, unused imports, style consistency, and common security patterns. * **Humans handle judgment:** Design, architecture, system-level decisions where domain context is irreplaceable. That split is what makes the workflow sustainable. When a human reviewer opens a PR and the mechanical issues have already been flagged, they get to spend their attention on the questions worth asking. [CodeRabbit](https://www.coderabbit.ai/), the AI code review platform, runs this first pass everywhere developers work: pull requests, [IDEs (VS Code, Cursor, Windsurf)](https://www.coderabbit.ai/blog/how-we-built-our-ai-code-review-tool-for-ides), CLI, and mobile [code review through Slack](https://www.coderabbit.ai/agent). Every PR gets a walkthrough summary and a run of 40+ bundled linters and SAST tools inline before a human sees it. The standards-enforcement angle matters just as much. Teams encode their conventions once into CodeRabbit Learnings, Code Guidelines, or path-based rules, and CodeRabbit applies them during pull request reviews. The rules live in the review process instead of a Notion page nobody reads. Sustainable code quality comes from treating reviewer attention as a finite resource and building systems that protect it. The work happens in layers: pre-commit checks before the PR opens, AI review handling the first pass, humans focused on design and architecture, CI/CD validating the merge. Each layer catches what the previous one missed, and the combination keeps the team shipping quality code without grinding through the people who write it. Ready to make code review sustainable for your team? [Get a 14-day free trial](https://app.coderabbit.ai/login?free-trial=) of CodeRabbit today.

FluxNinja joins CodeRabbit

Aravind Putrevu — Sat, 16 Mar 2024 00:00:00 GMT

We are excited to announce that CodeRabbit has acquired [FluxNinja](https://www.fluxninja.com/), a startup that provides a platform for building scalable generative AI applications. This acquisition will allow us to ship new use cases at an industrial pace while sustaining our rapidly growing user base. FluxNinja's Aperture product provides advanced rate and concurrency limiting, caching, and request prioritization capabilities that are essential for reliable and cost-effective AI workflows. Since our launch, [Aperture's open-source](https://github.com/fluxninja/aperture) core engine has been critical to our infrastructure. Our initial use case centered around [mitigating aggressive rate limits](https://blog.coderabbit.ai/blog/coderabbit-openai-rate-limits) imposed by OpenAI, allowing us to prioritize paid and real-time chat users during peak load hours while queuing requests from free users. Further, we used Aperture's [caching and rate-limiting capabilities](https://blog.coderabbit.ai/blog/how-we-built-cost-effective-generative-ai-application) to manage costs, which in turn allowed us to offer open-source developers a fully featured free tier by minimizing abuse. These capabilities allowed us to scale our user base without ever putting up a waitlist and at a price point that is sustainable for us. With Aperture's help, CodeRabbit has scaled to over 100K repositories and several thousand organizations under its review in a short period. We started CodeRabbit with a vision to build an AI-first developer tooling company from the ground up. Building enterprise-ready applied AI tech is unlike any other software engineering challenge of the past. Based on our learnings while building complex workflows, it became apparent that we need to invest in a platform that can solve the following problems: * **Prompt rendering:** Prompt design and rendering are akin to responsive web design. Web servers render pages based on the screen size and other parameters. For example, on a mobile device, navigation bars are usually rendered as hamburger menus, making them easier for humans to use. Similarly, we need a prompt service that can render prompts based on the context windows of underlying models and prioritize the packing of context based on business attributes, making it easier for AI consumption. It's not feasible to include the entire repository, past conversations, documentation, learnings, etc., in a single code review prompt because of the context window size limitations. Even if it were possible, AI models exhibit poor recall when performing inference on a completely packed context window. While tight packing may be acceptable for use cases like chat, it’s not for use cases like code reviews that require accurate inferences. Therefore, it's critical to render prompts in such a way that the quality of inference is high for each use case while being cost-effective and fast. In addition to packing logic, basic guardrails are also needed, especially when rendering prompts based on inputs from end users. Since we provide a free service to public repositories, we have to ensure that our product is not misused beyond its intended purpose or tricked into divulging sensitive information, which could include our base prompts. * **Validation and quality checks:** Generative AI models consume text and output text. On the other hand, traditional code and APIs require structured data. Therefore, the prompt service needs to expose a RESTful or gRPC API that can be consumed by the other services in the workflow. We touched upon the rendering of prompts based on structured requests in the previous point, but the prompt service also needs to parse and validate responses into structured data and measure the quality of the inference. This is a non-trivial problem, and multiple tries are often required to ensure that the response is thorough and meets the quality bar. For instance, we found that when we pack multiple files in a single code review prompt, AI models often miss hunks within a file or miss files altogether, leading to incomplete reviews. * **Observability:** One key challenge with generative AI and prompting is that it's inherently non-deterministic. The same prompt can result in vastly different outputs, which can be frustrating, but this is precisely what makes AI systems powerful in the first place. Even slight variations in the prompt can result in vastly inferior or noisy outputs, leading to a decline in user engagement. At the same time, the underlying AI models are ever-evolving, and the established prompts drift over time as the models get regular updates. Traditional observability is of little use here, and we need to rethink how we classify and track generated output and measure quality. Again, this is a problem that we have to solve in-house. While FluxNinja's Aperture project was limited to solving a different problem around load management and reliability, we found that the underlying technology and the team's expertise were a perfect foundation for building the AI platform. Prompt engineering is in its nascent stage but is emerging as a joystick for controlling AI behavior. Packing the context window with relevant documents (retrieval-augmented generation, aka RAG) is also emerging as the preferred way of providing proprietary data compared to fine-tuning the model. Most AI labs focus on increasing the context window rather than making fine-tuning easier or cheaper. Despite the emergence of these clear trends, applied AI systems are still in their infancy. None of the recent AI vendors seem to be building the "right" platform, as most of their focus has been on background/durable execution frameworks, model routing proxies/gateways, composable RAG pipelines, and so on. Most of these approaches fall short of what a real-world AI workflow requires. The right abstractions and best practices will still have to appear, and the practitioners themselves will have to build them. AI platforms will be a differentiator for AI-first companies, and we are excited to tackle this problem head-on with a systems engineering mindset. We are excited to have the FluxNinja team on board and to bring our users the best-in-class AI workflows. We are also happy to welcome [Harjot Gill](https://www.linkedin.com/in/harjotsgill/), the founder of FluxNinja, and the rest of the team to CodeRabbit.

How we built a cost-effective Generative AI application

Aravind Putrevu — Fri, 22 Dec 2023 00:00:00 GMT

Since its inception, CodeRabbit has experienced steady growth in its user base, comprising developers and organizations. Installed on thousands of repositories, CodeRabbit reviews several thousand pull requests (PRs) daily. We have [previously discussed](https://blog.coderabbit.ai/blog/coderabbit-openai-rate-limits) our use of an innovative client-side request prioritization technique to navigate OpenAI rate limits. In this blog post, we will explore how we manage to deliver continuous, in-depth code analysis cost-effectively, while also providing a robust, free plan to open-source projects. ## CodeRabbit's Product Offering and LLM Consumption CodeRabbit is an AI-first PR Review tool that uses GPT APIs for various functionalities. CodeRabbit offers the following tiers of service: * **CodeRabbit Pro:** A paid service providing in-depth code reviews for private repositories. It's priced according to the number of developers, starting with a full-featured 7-day free trial. * **CodeRabbit for Open Source:** A free service offering in-depth code reviews for open source (public) repositories. * **CodeRabbit Free:** A free plan for private repositories, providing summarization of code changes in a PR. Our vision is to offer an affordable, AI-driven code review service to developers and organizations of all sizes while supporting the open-source community. We are particularly mindful of open-source projects, understanding the challenges in reviewing community contributions. Our goal is to reduce the burden of code reviews for open-source maintainers by improving submission quality before the review process begins. CodeRabbit's review process is automatically triggered when a PR is opened in GitHub or GitLab. Each review involves a complex workflow that builds context and reviews each file using large language models (LLMs). Code review is a complex task that requires an in-depth understanding of the changes and the existing codebase. High-quality review comments necessitate state-of-the-art language models such as gpt-4. However, these models are significantly more expensive than simpler models, as shown by the [10x-30x price difference](https://openai.com/pricing) between gpt-3.5-turbo and gpt-4 models. ![How we built a cost-effective Generative AI application](https://framerusercontent.com/images/y2qM0vieM6oWHGW7iILbR8rbQ.png) The gpt-4 model is 10-30x more expensive than the gpt-3.5-turbo model. Our primary cost driver is using OpenAI's API to generate code review comments. We will share our cost optimization strategies in the following sections. Without these optimizations, our free offering to open-source projects would not be feasible. Let's take a look at the strategies that helped us optimize the cost and improve user experience. ## 1\. Dual Models: Summarize and Triage Using Simpler Models For less complex tasks such as summarizing code diffs, simpler models such as gpt-3.5-turbo are adequate. As an initial optimization, we use a mix of models, as detailed in [our earlier blog post](https://blog.coderabbit.ai/blog/coderabbit-deep-dive). We use gpt-3.5-turbo to compress large code diffs into concise summaries, which are then processed by gpt-4 for reviewing each file. This dual-model approach significantly reduces costs and enhances review quality, enabling us to manage PRs with numerous files and extensive code differences. Additionally, we implemented triage logic to skip trivial changes from the review process. We use the simpler model to classify each diff as either trivial or complex, as part of the same prompt used for code diff summarization. Low-risk changes such as documentation updates, variable renames, and so on, are thus excluded from the thorough review process. This strategy has proven effective, as simpler models can accurately identify trivial changes. By using this dual-model approach for summarization and filtering out trivial changes, we save almost 50% on costs. ## 2\. Rate-limiting: Enforcing Fair Usage Upon launching our free service for open-source projects, we noticed individual developers using it as a coding co-pilot by making hundreds of incremental commits for continuous feedback. CodeRabbit, designed for thorough code reviews unlike tools such as GitHub Copilot, incurs high costs when used in this manner. Therefore, we implemented hourly rate limits on the number of files and commits reviewed per user to control excessive usage without compromising user experience. These limits vary across different product tiers. For example, we set more aggressive limits for open-source users compared to trial and paid users. To implement these rate limits, we evaluated various options for serverless environments. We opted for [FluxNinja Aperture](https://fluxninja.com/) for its simplicity and policy sophistication. We were already using Aperture for managing [OpenAI rate limits](https://blog.coderabbit.ai/blog/coderabbit-openai-rate-limits), making it a natural choice for our rate-limiting needs as well. In FluxNinja Aperture, policies are decoupled from application logic through labels, enabling new policy additions without altering application code. We apply labels in FluxNinja Aperture, wrap the review workload with its SDK, and write policies that enforce limits on those labels. For example, we enforce a limit of 3 reviews per hour (1 review every 20 minutes) for open-source users, allowing a burst of 2 back-to-back reviews, as shown in the screenshots below. ![How we built a cost-effective Generative AI application](https://framerusercontent.com/images/1etMQYd13gki6CPITdyj6YXBPI.png) Integration with FluxNinja Aperture SDK *Rate-limiting commits per hour for open-source users* ![How we built a cost-effective Generative AI application](https://framerusercontent.com/images/itoaYJdM5RE6at6nxflhZKd88mk.png) *Wait time feedback to the user in a comment* Given the high cost and capacity constraints of state-of-the-art models such as gpt-4, rate-limiting is an essential requirement for any AI application. By implementing fair-usage rate limits, we are saving almost 20% on our costs. ![How we built a cost-effective Generative AI application](https://framerusercontent.com/images/ncC7v5MVxXsKGAOezXEVyd7SJbM.png) *Rate limit metrics for open-source users* ## 3\. Caching: Avoid Regenerating Similar Review Comments We believe that building user habits around AI involves seamlessly augmenting existing workflows. Therefore, AI code reviews must be continuous: they should trigger as soon as a PR is opened and incrementally update the summary and generate review comments as more commits are added. However, this approach can become expensive and generate repetitive feedback, as similar review comments are regenerated for each commit. We observed that most incremental commits involve minor adjustments or bug fixes in the initial implementation. To address this, we implemented a caching layer to avoid regenerating similar review comments for incremental commits. Fortunately, Aperture also provides a simple caching mechanism for summaries from previous commits, using the same API call where we implemented rate limits. During each incremental review, we use the simpler model for a semantic comparison of the code changes described in both summaries. If the changes are similar, we skip the review for those files to prevent regenerating similar review comments. This method differs from vector similarity-based caching techniques, as we use an LLM model for comparing summaries. Vector similarity-based approaches wouldn't be effective in our case, as the summaries require semantic comparison. We have integrated this method into the same prompt used for code diff summarization and triage. By using the more cost-effective gpt-3.5-turbo model as an advanced similarity filter before invoking the more expensive gpt-4 model for the same file, we have saved almost 20% of our costs by avoiding the generation of similar review comments. ## Conclusion In this blog post, we briefly discussed how state-of-the-art LLMs such as gpt-4 can be expensive in production. We also shared our strategy of using a combination of simpler models, rate limits, and caching to optimize operational costs. We hope our experiences can assist other AI startups in optimizing their costs and developing cost-effective AI applications.

Squeezing water from stone - managing OpenAI rate limits with request prioritization

Aravind Putrevu — Sun, 22 Oct 2023 00:00:00 GMT

Since CodeRabbit launched a couple of months ago, it has received an enthusiastic response and hundreds of sign-ups. CodeRabbit has been installed in over 1,300 GitHub organizations and typically reviews more than 2,000 pull requests per day. Furthermore, usage continues to flourish; we are experiencing healthy week-over-week growth. While this rapid growth is encouraging, we've encountered challenges with OpenAI's stringent rate limits, particularly for the newer `gpt-4` model that powers CodeRabbit. In this blog post, we will delve into the details of OpenAI rate limits and explain how we leveraged the [FluxNinja Aperture](https://www.fluxninja.com/) load management platform to ensure a reliable experience as we continue to grow our user base. ## Understanding OpenAI rate limits OpenAI imposes [fine-grained rate limits](https://platform.openai.com/docs/guides/rate-limits/overview) on both requests per minute and tokens per minute for each AI model they offer. In our account, for example, we are allocated the following rate limits: ![Squeezing Water from Stone - Managing OpenAI Rate Limits with Request](https://framerusercontent.com/images/WCNgIwtyOdTtC8iOFvbU3axD8.png) We believe that the rate limits are in place for several reasons and are unlikely to change in the near future: * Advanced models such as `gpt-4` are computationally intensive. Each request can take several seconds or even minutes to process. For example, a 30-second response time is fairly typical for complex tasks. OpenAI sets these limits to manage aggregate load on their infrastructure and provide fair access to users. * The demand for AI has outstripped the supply of available hardware, particularly the GPUs required to run these models. It will take some time for the industry to meet this exploding demand. ## CodeRabbit's OpenAI usage pattern and challenges CodeRabbit is an AI-driven code review application that integrates with GitHub or GitLab repositories. It analyzes pull requests and provides feedback to the developers on the quality of their code. The feedback is provided in the form of comments on the pull request, allowing the developers to enhance the code based on the provided suggestions in the follow-up commits. ![Squeezing Water from Stone - Managing OpenAI Rate Limits with Request](https://framerusercontent.com/images/Y83rpWAtskEjGPenr3dNOUm6mY.png) CodeRabbit employs a combination of the `gpt-3.5-turbo` and `gpt-4` family of models. For simpler tasks such as summarization, we use the more economical `gpt-3.5-turbo` model, whereas intricate tasks such as in-depth code reviews are performed by the slow and expensive `gpt-4` model. Our usage pattern is such that each file in a [pull request is summarized and reviewed concurrently](https://www.coderabbit.ai/blog/coderabbit-deep-dive). During peak hours or when dealing with large pull requests (consisting of 50+ files), we began to encounter `429 Too Many Requests` errors from OpenAI. Even though we had a retry and backoff mechanism, many requests were still timing out after multiple attempts. Our repeated requests to OpenAI to increase our rate limits were met with no success. To mitigate these challenges, we cobbled together a makeshift solution for our API client: * We set up four separate OpenAI accounts to distribute the load. * We implemented an API concurrency limit on each reviewer instance to cap the number of in-flight requests to OpenAI. * We increased the backoff time for each retry and increased the number of retries. OpenAI's rate limit headers were not helpful in determining the optimal backoff times, as the [headers](https://platform.openai.com/docs/guides/rate-limits/rate-limits-in-headers) were outdated by tens of seconds and did not consider the in-flight requests. * We transitioned from a function-based serverless framework to a containerized environment to benefit from extended timeout capabilities and ensure that instances would not be terminated while requests were in the retry-backoff loop. Although these adjustments provided temporary relief, the challenges resurfaced as the load increased within a few days. We were doing much guesswork to figure out the "right" number of concurrent requests, backoff times, maximum retries, and so on. Complicating matters further, we added a chat feature that allows users to consult the CodeRabbit bot for code generation and technical advice. While we aimed for real-time responses, the back-off mechanisms made reply time unpredictable, particularly during peak usage, thereby degrading the user experience. We needed a better solution, one that could globally manage the rate limits across all reviewer instances and prioritize requests based on user tiers and the nature of requests. ## FluxNinja Aperture to the rescue We were introduced to the [FluxNinja Aperture](https://www.fluxninja.com/) load management platform by one of our advisors. [Aperture](https://github.com/fluxninja/aperture) is an open-source load management platform that offers advanced rate-limiting, request prioritization, and quota management features. Essentially, Aperture serves as a global token bucket, facilitating client-side rate limits and business-attribute-based request prioritization. ### Implementing the Aperture TypeScript SDK in our reviewer service Our reviewer service runs on Google Cloud Run, while the Aperture Agents are deployed on a separate Kubernetes cluster (GKE). To integrate with the Aperture Agents, we employ Aperture's TypeScript SDK. Before calling OpenAI, we rely on Aperture Agent to gate the request using the `StartFlow` method. To provide more context to Aperture, we also attach the following labels to each request: * `model_variant`: This specifies the model variant being used (`gpt-4`, `gpt-3.5-turbo`, or `gpt-3.5-turbo-16k`). Requests and tokens per minute rate limit policies are set individually for each model variant. * `api_key`: This is a cryptographic hash of the OpenAI API key, and rate limits are enforced on a per-key basis. * `estimated_tokens`: As the tokens per minute quota limit is enforced based on the [estimated tokens for the completion request](https://platform.openai.com/docs/guides/rate-limits/reduce-the-max_tokens-to-match-the-size-of-your-completions), we need to provide this number for each request to Aperture for metering. Following OpenAI's [guidance](https://help.openai.com/en/articles/4936856-what-are-tokens-and-how-to-count-them), we calculate `estimated_tokens` as `(character_count / 4) + max_tokens`. Note that OpenAI's rate limiter doesn't tokenize the request using the model's specific tokenizer but relies on a character count-based heuristic. * `product_tier`: CodeRabbit offers both `pro` and `free` tiers. The `pro` tier provides comprehensive code reviews, whereas the `free` tier offers only the summary of the pull request. * `product_reason`: We also label why a review was initiated under the `pro` tier. For example, the reason could be that the user is a `paid_user`, `trial_user`, or an `open_source_user`. Requests to OpenAI are prioritized based on these labels. * `priority`: Requests are ranked according to a priority number provided in this label. For instance, requests from `paid_user` are given precedence over those from `trial_user` and `open_source_user`. The base priority is incremented for each file reviewed, enabling pull requests that are further along in the review process to complete more quickly than newly submitted ones. Additionally, chat messages are assigned a much higher priority compared to review tasks. ![Squeezing Water from Stone - Managing OpenAI Rate Limits with Request](https://framerusercontent.com/images/aCvZzngJXymwfYDVcMKq2XDKlM.png) Integration with Aperture TypeScript SDK ### Policy configuration in Aperture: Aligning with OpenAI's rate limits Aperture offers a foundational "blueprint" for [managing quotas](https://docs.fluxninja.com/reference/blueprints/quota-scheduling/base), comprising two main components: * **Rate limiter:** OpenAI employs a token bucket algorithm to impose rate limits, and that is directly compatible with Aperture's rate limiter. For example, in the tokens per minute policy for `gpt-4`, we have allocated a burst capacity of `40000 tokens` and a refill rate of `40000 tokens per minute`. The bucket begins to refill the moment the tokens are withdrawn, aligning with OpenAI's rate-limiting mechanism. This ensures our outbound request and token rates remain synchronized with OpenAI's enforced limits. * **Scheduler:** Aperture has a [weighted fair queuing](https://docs.fluxninja.com/concepts/scheduler/) scheduler that prioritizes the requests based on multiple factors, such as the number of tokens, priority levels, and workload labels. By fine-tuning these two components in Aperture, we can go as fast as we can with an optimal user experience while ensuring that we don't exceed the rate limits. Client-side quota management policies for gpt-4 ## Reaping the benefits During peak hours, we typically process tens of pull requests, hundreds of files, and chat messages concurrently. The image below shows the incoming token rate and the accepted token rate for the `gpt-4` tokens-per-minute policy. We can observe that the incoming token rate is spiky, while the accepted token rate remains smooth and hovers around `666 tokens per second`. This roughly translates to `40,000 tokens per minute`. Essentially, Aperture is smoothing out the fluctuating incoming token rate to align it with OpenAI's rate limits. ![Squeezing Water from Stone - Managing OpenAI Rate Limits with Request](https://framerusercontent.com/images/tVKawcfxMtGtulPpNuuHSUVxVM.png) The image below shows request prioritization metrics from the Aperture Cloud console during the same peak load period: ![Squeezing Water from Stone - Managing OpenAI Rate Limits with Request](https://framerusercontent.com/images/VXO8ccw5QJvUSCeam41bZRISrQ.png) In the upper left panel of the metrics, noticeable peaks indicate that some requests got queued for several minutes in Aperture. We can verify that the trial and free-tier users tend to experience longer queue times compared to their paid counterparts and chat requests. Queue wait times can fluctuate based on the volume of simultaneous requests in each workload. For example, wait times are significantly longer during peak hours than during off-peak hours. Aperture provides scheduler preemption metrics to offer further insight into the efficacy of prioritization. As observed in the lower panels, these metrics measure the relative impact of prioritization for each workload by comparing how many tokens a request gets preempted or delayed in the queue compared to a purely First-In, First-Out (FIFO) ordering. In addition to effectively managing the OpenAI quotas, Aperture provides insights into OpenAI API performance and errors. The graphs below show the overall response times for various OpenAI models we use. We observe that the `gpt-4` family of models is significantly slower compared to the `gpt-3.5-turbo` family of models. This is quite insightful, as it hints at why OpenAI's infrastructure struggles to meet demand - these APIs are not just simple database or analytics queries; they are computationally expensive to run. ![Squeezing Water from Stone - Managing OpenAI Rate Limits with Request](https://framerusercontent.com/images/Te7MPRH8HaivWTczZWpZ6YBlqrM.png) ## Conclusion Aperture has been a game-changer for CodeRabbit. It has enabled us to continue signing up users and growing our usage without worrying about OpenAI rate limits. Without Aperture, our business would have hit a wall and resorted to a waitlist approach, which would have undermined our traction. Moreover, Aperture has provided us with valuable insights into OpenAI API performance and errors, helping us monitor and improve the overall user experience. In the realm of generative AI, we are dealing with a fundamentally different nature of API dynamics. Performance-wise, these APIs are an order of magnitude slower than traditional APIs. We believe that request scheduling and prioritization will become a critical component of the AI infrastructure stack, and with Aperture, FluxNinja is well-positioned to be the leader in this space. As CodeRabbit continues to build and add additional components such as vector databases, which are also computationally expensive, we are confident that Aperture will continue to help us offer a reliable experience to our users.