Same model, six code reviewers, 38% to 81% of the bugs

TL;DR

Which AI code review tool you run mattered far more for finding bugs than moving it to a newer model. Before you pay for an upgrade, give your reviewer the pull request description and a higher effort setting.

A pull request to Microsoft’s Go port of the TypeScript compiler taught it to read JSDoc @import comments. Its tests passed. It also made type-checking crash on a file as ordinary as /** @import { Foo } from './types' */ export const x = ...¹⁴.

I gave that PR to six AI code reviewers, all running the same model, GPT-6.1 Sol. Two reported the crash, mine and one other. One got halfway. Three missed it.

The whole test went the same way. Across 145 PRs, the six reviewers on that one model found between 38% and 81% of the same 394 bugs.

Then I kept the reviewers fixed and changed the model. On 40 of those PRs, I moved all six from GPT-6 Sol to the newer GPT-6.1 Sol. None of the six found more than 4 percentage points more, and one found 10 points fewer.

The reviewer at 81% is mine. I build QuickReview at Quickchat AI, and I designed and ran this test¹². So here is what a skeptic would dig up:

• Claude agents built the answer key, the list of bugs every review is scored against, partly from QuickReview’s own earlier reviews. No human has checked the key or the AI judge that does the scoring².
• QuickReview cost the most: $1.41 a PR, against 7 to 59 cents for the other five. It also posted the most comments.
• Most of its lead is in medium- and low-severity bugs.

Over the three days after this run I tuned QuickReview on these same PRs, which took it to 90%. That number stays out of the headline. The 81% comes from the build before that tuning, on PRs its design had never looked at⁹.

The finding does not depend on my tool. Take QuickReview out, and the other five still spread from 38% to 69% on one model.

On one model, six reviewers found 38% to 81% of the same bugs

The PRs come from AACR-Bench, a public benchmark of real pull requests from open-source projects. Its own answer key is a set of review comments. When agents checked those comments against the code, about one in eight was part of a real bug.

So Claude agents built a new key: 520 bugs across the 185 PRs that still run². Each bug was checked against the code and given a severity:

• high: would block a merge
• medium: fix before merging
• low: real, but can wait

The charts and the notes say “defect” where I say “bug”.

The key, every review and every verdict are public as a dataset on Hugging Face (DOI 10.57967/hf/11598), so you can recount what follows¹³.

The headline uses 145 of those PRs, with 394 bugs. I left out the other 40 because I had tuned earlier QuickReview builds on them⁹. Every reviewer ran once on GPT-6.1 Sol¹. Each third-party tool ran in the configuration that had found the most in a trial of its documented options. That trial used 40 of these PRs, under a rule fixed before it started⁵.

A hostile reader would run three checks on that spread.

The key leans toward QuickReview, and the order survives without that. 250 of the 394 bugs were raised by a blind hunt through the code or by one of the benchmark’s own comments, so no reviewer’s comment put them in the key. On those 250, the six found 46% to 86%, in the same order².

On merge-blocking bugs, four of the six were close. There are 56 high-severity bugs. QuickReview found 54 of them, Codex CLI 52, pi-review’s three-reviewer panel 50 and OpenCode 48. open-code-review found 41.5 and PR-Agent 35.5, where a partly found bug counts as half. QuickReview’s lead over pi-review’s panel, the next best overall, was 51 bugs, and 45 of them were low severity.

Tuning on these PRs added 9 points, and some of that carried over. After the tuning round QuickReview found 90% here and 89% of all 520. On the 20 PRs that round never used, it rose from 70% to 83%. On those 20, pi-review’s panel found 72% and Codex CLI 53%⁹. Before the round QuickReview had been level with pi-review’s panel there.

Claude Code’s local /code-review is outside the six because it runs on Anthropic’s models. On Claude Opus 5.5, at its default effort, version 2.1.283 found 63% of the same 394 bugs for $0.21 a PR³.

Four of the six also ran on two other code review benchmarks, Martian’s and Qodo’s, again on GPT-6.1 Sol. On both, QuickReview’s tuned build matched more of the answer key than open-code-review, OpenCode and PR-Agent. On both, it also had the lowest precision of the four, meaning the smallest share of its comments matched the key. On Martian’s own score it trailed open-code-review. Its tuning had used 20 of Martian’s 50 PRs and 59 of Qodo’s 580 issues⁶.

A newer model made the reviewers more accurate and found no more bugs

The model comparison uses the 40 trial PRs, which are among the 145 and hold 115 bugs. The trial that picked each tool’s configuration ran on GPT-6 Sol, and the main runs used its successor, GPT-6.1 Sol. So on those 40, each third-party tool has one run on each side of the upgrade, in the same configuration⁵.

PR-Agent gained 4 points and pi-review’s panel 2. Codex CLI did not move, open-code-review lost 2 and OpenCode lost 10. On the newer model the six still stood 51 points apart, from 34% to 85%.

The upgrade did buy accuracy. The share of comments the judge graded correct rose for all five third-party tools:

If your reviewer’s problem is wrong comments, a newer model may fix it. If its problem is missed bugs, these runs say to look elsewhere.

QuickReview is the sixth line in the chart. Its default preset found 85% of those 115 bugs on the newer model, against 86% and 90% in two runs on the older one. Its low preset rose from 67% to 75%. Both comparisons cross builds a few days apart, so I read them as no clear change⁴.

The model can matter a great deal when the step is bigger. With a small model, GPT-6 Luna, in every role, QuickReview found 28% of all 520 bugs, against 84% on GPT-6 Sol. And on the 40 PRs I had tuned it on, it found 95% on Claude Opus 5.5 against 70% on GPT-6.1 Sol⁴.

So the claim here is a narrow one: for the upgrade from one strong model to its successor, the reviewer mattered more. The next post runs one reviewer on three models. The three found about as many bugs, missed different ones and wrote different reviews.

Two settings were worth 10 to 11 points each

The same trial shows which parts of a reviewer mattered. Before the main runs, each third-party tool ran once in every configuration its documentation ships or recommends, on those same 40 PRs and on GPT-6 Sol. Pairs of those runs differ in one setting, so the gap between the two shows what that setting was worth⁵.

The PR description. Passing the title and description to Codex CLI and to OpenCode lifted each by 10 points at no extra cost. A description says what the change is meant to do, and a reviewer needs that to notice a change that does something else.

One kind of bug shows this most clearly. The key counts as a bug a fix the PR claims that does not hold everywhere, and it has 57 of these. QuickReview, which read the description, found 89% of them. Codex CLI, which did not, found 46%³. The description is no guarantee, though: pi-review and open-code-review moved by only a point or two.

The effort setting. This setting controls how much reasoning the model spends. High effort took Codex CLI from 44% to 56%, for $0.24 a PR instead of $0.19. pi-review gained 10 points from medium to high thinking, the level it ships with. QuickReview’s default preset differs from its low preset only in reasoning effort, and it found 85.7% against 67.4%. In the opening example, QuickReview at its low preset missed the crash that its default found.

More reviewers. This helped least for the money. pi-review’s three-reviewer panel found 6 points more than one reviewer at the same effort, for $0.46 a PR instead of $0.17.

Each row rests on 40 PRs and one run. Two runs of one QuickReview build differed by 4 to 8 points on sets of 10 PRs, so I trust the 10-point moves and not the single-digit ones¹¹.

One cheap agent matched QuickReview on 20 hard pull requests

The cost of a reviewer is a poor guide to what it finds, and my own tool is the example.

For my October post I took the 20 PRs of Martian’s benchmark where QuickReview had done worst on two other models. On those 20 I ran Claude Code 2.1.303’s local /code-review high against QuickReview, both on Claude Opus 5.5⁷. Claude Code ran one agent with no separate verification step. QuickReview ran its full set of passes, each one a review agent with a single concern, and a validator that checks every finding again.

Each matched 22 of the 27 serious issues in Martian’s answer key. Claude Code cost $0.33 a PR and QuickReview $4.61.

QuickReview also made more claims that matched nothing in the key: 16.2 a PR, against 9.8. That pulled its result on Martian’s own score down to 35.0, against 43.2.

The selection cuts both ways. These 20 were picked because QuickReview had done badly on them, which favours any other reviewer. In the other direction, 7 of them were among the PRs its tuning had used⁷.

A second measure went QuickReview’s way. Two AI judges read both reviews whole and preferred QuickReview’s in 29 of 40 verdicts. That margin came mostly from one of the two judges, as the first post in this series examines. On the count of answer-key issues matched, one agent at high effort was enough.

What the extra bugs cost

Back on the 145 PRs, QuickReview cost about 9 times as much as Codex CLI and 5 times as much as pi-review’s panel⁸. The table shows what the extra money bought.

The comments are a cost too. QuickReview posted 5.6 a PR, against 1.9 for Codex CLI. The judge graded 61% of QuickReview’s comments worth raising before merge, against 90% of Codex CLI’s. It graded a third of QuickReview’s true and minor.

A QuickReview review also took longer, a median 9.5 minutes against 1.9. The tuned build costs more again: $2.13 a PR here and 7.2 comments.

Whether that pays depends on what a missed bug costs you. Rigby and Bird studied review on the Google-led Android and Chromium OS projects, at Microsoft and at AMD. They wrote that “any reduction in the number of reviewers that does not lead to a reduction in the number of defects found will result in cost savings”¹⁰. The same holds for review passes. An extra pass pays only when the bugs it alone finds would have shipped and cost more than the pass.

What to change in your own setup

Before switching models, I would change the reviewer around the one I have, in this order.

1. Give it the PR description. Check that your tool actually receives the title and description. A review command that is handed only a diff does not. Add the repository’s written rules if it can take them. This was worth 10 points to two tools and cost nothing.
2. Raise the effort setting. For Codex CLI that was 11 points for about 6 cents a PR.
3. Measure before adding agents. Three reviewers instead of one bought pi-review 6 points for 2.8 times the cost. And one Claude Code agent matched my multi-pass reviewer on 20 hard PRs.
4. Count the passes that ran. Two reviewers skipped steps that only a prompt asked for: Claude Code 2.1.303’s multi-agent /code-review max with Claude Sonnet 5.5 agents, and QuickReview’s own session on Claude Sonnet 5.5⁷. The code that runs the reviewer should launch each planned pass and fail the review when one is missing.
5. Upgrade the model for accuracy. Expect fewer wrong comments from a newer model. In these runs it did not find more bugs.

Then price it on your own history. Collect merged PRs that later needed a fix, run both setups on them on one model, and divide the extra spend by the extra bugs found. If only merge-blocking bugs matter to you, the cheap end is already good: Codex CLI at high effort found 52 of the 56 high-severity bugs for 15 cents a PR.

How QuickReview fits, and what it costs

QuickReview is in beta. It is open source under the MIT licence, runs on your own infrastructure under your own keys, and has reviewed a busy commercial repository continuously since mid-August 2026. A review works like this:

• Plain code, with no model involved, plans its passes.
• Findings about one root cause merge into one comment.
• A validator re-derives each candidate from the code.
• One review posts, with a severity on every finding and a verdict.

Install and run it with:

npm i -g @quickchatai/quickreview
qcr review

In pi, pi install npm:@quickchatai/pi-quickreview adds it, and /qcr-review --preview shows the plan and the price without running anything. dry_run = true is the shipped default, so a fresh install posts nothing.

QuickReview costs more and takes longer than every other tool here. On Martian’s and Qodo’s benchmarks it had the lowest precision of the four tools that ran.

Its quick preset is the low preset above. That preset found 72% of the 394 bugs for $0.91 a PR, against 81% for $1.41 at the default. Builds after the one frozen on 4 October changed some cost settings, so today’s cost may differ.

Limitations

• This is a vendor’s test of its own product against competitors¹².
• Each configuration ran once.
• One AI judge scored each benchmark. For the data release, another judge, from a different model family, re-scored the five third-party tools and QuickReview’s tuned build without seeing the code. Over all 185 PRs it moved none of them by more than 0.4 points and kept their order, and it graded fewer of their comments correct. It did not re-score the build behind the 81%. No human has checked a judge or the AACR-Bench key, which Claude agents built partly from QuickReview’s own earlier reviews².
• The 145 PRs were new to QuickReview’s design when its 1 October build ran. Every later number for the tuned build comes from PRs it was tuned on⁹.
• The model comparison rests on 40 PRs and on one pair of models from one maker.
• QuickReview read the PR description and package sources and spent more than any other tool. No equal-budget comparison was run³.

The data is public, with the scripts that recompute the papers’ numbers from it¹³.

Frequently asked questions

Does the model or the reviewer around it matter more for AI code review?

In these runs, the reviewer. On GPT-6.1 Sol, six reviewers found 38% to 81% of the same 394 verified bugs in 145 pull requests. Without my own QuickReview, the other five still spread from 38% to 69%. Moving them from GPT-6 Sol to the newer GPT-6.1 Sol changed what each found by at most 4 points up and 10 down. A much smaller model did matter: GPT-6 Luna cut QuickReview to 28%.

How much more does a multi-pass AI code reviewer cost than a single-pass review command?

On GPT-6.1 Sol, QuickReview cost about 9 times as much per pull request as Codex CLI ($1.41 against $0.15) and 5 times as much as pi-review’s three-reviewer panel ($0.29). That came to about $10 and $27 in model spend per extra high- or medium-severity bug it found.

Is it fair to compare QuickReview with other code reviewers on the same model?

It compares whole tools, and one of them is mine. QuickReview reads the pull request description and spends far more, and each third-party tool ran the best of its documented configurations. AI agents built the answer key partly from QuickReview’s earlier reviews. No equal-budget comparison was run.

When is a cheaper AI code review enough?

When only merge-blocking bugs matter. On GPT-6.1 Sol, Codex CLI at high effort found 52 of 56 high-severity bugs for 15 cents a pull request, against 54 for QuickReview at $1.41.

Does a newer model make an AI code reviewer find more bugs?

In these runs it made reviewers right more often. Five third-party reviewers moved from GPT-6 Sol to GPT-6.1 Sol on 40 pull requests. Their share of correct comments rose by 2 to 23 points. Their share of bugs found moved by less than 5 points for four of them and fell by 10 for OpenCode.

Did this test Anthropic’s managed Code Review or the local /code-review command?

Only the local /code-review command in the Claude Code CLI, versions 2.1.283 and 2.1.303. Anthropic’s managed Code Review service and Ultrareview were not tested, and neither was any hosted review service from OpenAI.

Related guides

• What Claude Code’s multi-agent code review costs and finds
• AI judges agree on known bugs and split on which code review is better
• My AI code reviewer kept missing bugs in the very lines it commented on
• Same code reviewer, three models, three different reviewers

Method and data notes

These notes hold the method, definitions and caveats behind every number above. The body links to each note by its number. The notes say “defect” where the body says “bug”. The method and every table are in two papers, Measuring whether a code review finds the real defects and QuickReview: a multi-pass automated code reviewer.

1. Every configuration, with its billed model

• Claude Code is the local command. Every Claude Code result here is the /code-review command that ships with the Claude Code CLI, never Anthropic’s managed Code Review service or Ultrareview. Martian’s leaderboard entry named claude-code came from the managed service, so it appears nowhere in this post. Codex CLI likewise ran its local codex exec review, and no hosted review service from either company was tested. The judge read Claude Code’s whole answer as well as the parsed findings, so its defect recall depends little on our parser.
• Run dates. The earlier study’s runs, Claude Code 2.1.283 and Codex CLI 0.158.0 among them, ran from 28 to 30 September 2026, the selection runs on 30 September, Codex CLI 0.159.2 and the other picks on GPT-6.1 Sol from 30 September to 1 October, the frozen build on 4 October, and Claude Code 2.1.303 on the 20 hard Martian pull requests from 19 October.
• QuickReview’s passes. On the GPT-6 Sol default run a review executed a mean of 11.2 passes including the validator (median 11, range 4 to 24), planned by code from path patterns, content patterns in the added lines, sizes and the length of the PR description. On that run the findings records hold 2,959 candidates, of which 1,456 were merged as duplicates of another finding, 1,034 posted, 164 demoted and posted, 207 killed by the validator, 34 folded and 64 withheld by the main session.
• Words used here. Recall is the share of the key’s defects a review finds, with a partly found defect counted as half. A pass is one single-concern review agent; QuickReview’s papers call these lenses. The frozen build is QuickReview’s build of 4 October 2026, the one its papers report and the one the body calls the tuned build; the 1 October build is the one the benchmark paper’s final comparison ran, three days older; the September builds are earlier ones. The 145 pull requests of the headline are the set the papers call R145, and the 40 of the model comparison are the set they call S40, which lies inside it.

2. The AACR-Bench key and the judge

• The benchmark. AACR-Bench holds 200 pull requests from 50 public repositories. Its harness scores 196, and 11 of those could not be run because GitHub no longer serves their reviewed head commit, which leaves 185 pull requests in ten languages.
• Why a new key. AI agents checked AACR-Bench’s reference comments against the code on 145 of the pull requests. Only 142 of the 1,107 comments there (12.8%) turned out to be part of a real defect. Over all 185, only 162 of the 520 verified defects (31%) have a reference among their sources.
• The key. 520 defects, 90 high, 145 medium and 285 low; a partly found defect counts half. Claude agents (Claude Code subagents on Opus 5.5) built and critiqued it, and no human has checked any part of it.
• It was built partly from reviewers’ comments. QuickReview’s default on GPT-6 Sol was the largest single source of the first-version key on those 145 pull requests, raising 334 of its 379 defects (88%) and 29 that no other source raised in that first version. On the 369 defects that the expert hunt or an AACR-Bench reference raised, which would be in the key had no reviewer’s comment been a candidate, QuickReview’s 1 October build on GPT-6.1 Sol found 81%, Claude Code on Opus 5.5 70% and Codex CLI on GPT-6.1 Sol 60%. On the 145 pull requests of the headline, 250 of the 394 defects have the hunt or a reference among their sources. Of those 250, QuickReview’s 1 October build found 215 (86%), pi-review’s panel 181 (72%), open-code-review 165.5 (66%), Codex CLI 162 (65%), OpenCode 154 (62%) and PR-Agent 116 (46%), all on GPT-6.1 Sol. Claude Code on Opus 5.5 found 186.5 (75%). Without the 28 defects of the final key whose only sources are QuickReview’s earlier default and low runs, the order of all eight reviewers is unchanged.
• Pair by pair. In the GPT-6 Sol pair, QuickReview’s row is the run whose comments raised those 334 defects, and Codex CLI’s comments from the same study were checked later with three other reviewers’ and could only add defects, 15 in all. A re-run of QuickReview’s default on GPT-6 Sol on 29 September, made after three changes to the reviewer and with comments that were never candidates for the key, found 335.5 defects on those 145 pull requests, the same as the original run. On the 369, that pair stands at 86% against 53%, a 33-point gap against 36 on the whole key. The keys for the Opus pair’s 40 pull requests were built during QuickReview’s tuning from a blind hunt, the references and the comments of QuickReview’s earlier runs and, on 30 of them, open-code-review’s; Claude Code’s comments, checked later, added no defect there. In the headline pair on GPT-6.1 Sol neither reviewer’s comments were candidates, so a real defect that only one of them found is credited to neither. The papers report the 369-defect check for the 1 October build; computed the same way from the dataset, the frozen build finds 332 of the 369 (90%) against Codex CLI’s 222 (60%).
• The judge. GPT-6 Sol at high reasoning judged every AACR-Bench review in this post. In the earlier study a second judge from another family, Claude Opus 5.5, gave the same verdict on 3,593 of 3,640 reviewer-defect pairs (98.7%). That check covered the earlier study’s seven reviewers, the GPT-6 Sol pair and Claude Code’s run among them, and none of the GPT-6.1 Sol runs. Inside each pair both reviewers run on one model, so a judge’s preference for its own model family would apply to both.
• A third judge, without the code. For the data release DeepSeek V4.1 Flash, from a third model family, re-graded the five third-party reviewers of the headline, Claude Code and QuickReview’s frozen build. It had the primary judge’s prompt and each review, and no access to the code. Over all 185 pull requests every reviewer’s recall stayed within 0.4 points of the primary judge’s and in the same order, and the two judges gave the same verdict on 97.9% to 99.6% of each reviewer’s defects. On the 145, by my count from the dataset, the five third-party tools still run from 38% to 69% and the frozen build stays at 90%, and none moves by more than 2 defects. Comment grades are less stable. This judge graded fewer comments correct for six of the seven, by 1.2 to 9.3 points, so the accuracy figures in the body lean on the primary judge more than the recall figures do. QuickReview’s 1 October build, the 81% of the headline, was not re-graded, and no human has checked this judge either.
• Saying more. The judge credits a defect wherever a review states its essential point, as that defect’s own notes define it, so a vague comment near the right line does not count. In the final comparison, scoring a partly found defect as zero instead of half moved no reviewer by a point.
• AACR-Bench’s own score. On its reference metric every reviewer of the final comparison recovers 4.3% to 10.7% of the references; QuickReview’s 1 October build on GPT-6.1 Sol has the highest F1, 12.5% at a precision of 15.2%, against 6.9% to 11.6% for the others, Codex CLI’s 7.8% at 20.0% among them. The frozen build was not scored on it.

3. What differs inside each pair, besides the design

• Inputs. QuickReview, OpenCode and PR-Agent read the pull request’s title and body; pi-review, open-code-review, Codex CLI and Claude Code did not, because their benchmark runners have no channel for it or their selection runs did better without it. QuickReview also downloads the source archives of the npm and PyPI packages a pull request imports, at the versions its lockfiles pin, and unpacks them as plain text; nothing is installed or run. The key counts as a defect a fix the pull request claims that does not hold everywhere, and on the 57 such defects QuickReview’s 1 October build on GPT-6.1 Sol, which read the description, found 89% and Codex CLI, which did not, 46%.
• Effort. Claude Code was not put through the configuration protocol. It stayed at its published /code-review at the session default, medium, chosen to match the medium reasoning of QuickReview’s passes. Anthropic documents that at low and medium effort the review “reports only the findings it’s most confident in” and that higher levels broaden coverage. Its read-only git commands were denied on 46 of the 185 pull requests of that run, and the benchmark paper calls its 62% on all 185 “not a best-shot figure”.
• Budget and time. In the three pairs QuickReview cost 11 to 20 times the model maker’s command per pull request (note 8). In the benchmark paper’s final comparison its 1 October build on GPT-6.1 Sol cost 2.6 to 21 times each third-party tool and wrote 5.1 to 24 times its output tokens. Most of these differences favour QuickReview on defects found, and a strict equal-budget comparison was not done. On GPT-6.1 Sol the frozen build took a median 17.5 minutes a pull request and Codex CLI 2.1.
• The Opus pair. QuickReview on Opus 5.5 ran once, only on the 40 pull requests its September build was tuned on, and the paper calls the pair “a contrast, not a measured harness effect”. Those 40 are harder for every reviewer; Claude Code found 60% there and 63% on the other 145. No interval was computed for this pair.
• The GPT-6 Sol pair. Codex CLI ran as first shipped, at medium effort and without the PR text. On the 40 selection pull requests, high effort took it from 44% to 56%, so this pair overstates the gap a best-shot Codex CLI would leave on that model.

4. The model swap inside QuickReview

• 185 pull requests. With the design, the default preset and the reasoning levels fixed and the model behind every role changed, GPT-6 Sol found 436 and 437.5 defects in two runs (84%), GPT-6.1 Sol 409 (79%) and GPT-6 Luna 147 (28%), 39 of the 90 high-severity ones.
• The 40 pull requests of the Opus pair. Opus 5.5 found 119.5 (95%), GPT-6 Sol 100.5 and 102 (80% and 81%), GPT-6.1 Sol 88 (70%) and GPT-6 Luna 31.5 (25%). On the same 40, the frozen GPT-6.1 Sol build found 33 of 34 high, 28 of 31 medium and 44 of 61 low, 105 of 126 (83%).
• On the 40 pull requests of the model comparison. QuickReview’s default found 98.5 and 103 of the 115 defects in two runs on GPT-6 Sol (86% and 90%) and 98 (85%) on GPT-6.1 Sol; its low preset found 77.5 (67%) and then 86.5 (75%). The GPT-6 Sol runs are September builds and the GPT-6.1 Sol runs the 1 October build.
• Builds. The arms ran on builds committed between 27 and 29 September 2026: GPT-6 Sol on its original builds and on a re-run after three changes to the reviewer, GPT-6 Luna and Opus 5.5 on one build, and GPT-6.1 Sol on the 1 October build, which also needed a newer pi (0.99.1 against 0.87.1) and was built from a checkout with unsaved changes. A one-PR check ruled out a large cost change from the pi version and left a small effect on recall open.
• Less reasoning, same plan. With every role at low reasoning, QuickReview’s 1 October build found 70% on GPT-6.1 Sol at $0.88 a pull request against 79% at $1.36 for its default, a difference the report calls established; on GPT-6 Sol the September build’s gap was 66% against 84%.

5. How each third-party configuration was chosen

• The protocol. Each tool’s candidates were its published run plus the variants its own documentation ships or recommends, and the PR text where the tool can take it. Every candidate ran once on GPT-6 Sol on 40 pull requests drawn from the 145 outside QuickReview’s September tuning set, taken by the hash of a fixed seed and holding 115 defects (17 high). The rule, fixed before any selection run, picked the highest recall, with ties going to the higher share of correct comments. The picks then ran unchanged on GPT-6.1 Sol over all 185.
• Exceptions. Claude Code was not varied (note 3), and PR-Agent varied only its model, so its second candidate ran its shipped defaults on GPT-6.1 Sol. Not run: Claude Code’s /code-review at a higher level, pi-review at xhigh and Codex CLI at high effort with the PR text, and QuickReview never ran without the PR text. QuickReview’s rows on the 40 are reference points, not candidates.
• Caps and plumbing. PR-Agent’s shipped defaults allow at most three key issues and three suggestions per chunk within a 32,000-token prompt budget. open-code-review and PR-Agent reached the model through our loopback bridge, and no reviewer could search the web.
• What a pick is worth. A pick that is the best of two to four noisy runs is biased upward on the 40. On the other 145 pull requests, those not used for selection, the picks moved by −11 to +4 points and the order was the same as on all 185. In the paper’s words, “Giving every tool its best shot moved the field up; it did not close the gap.”
• The move to GPT-6.1 Sol. Each pick was re-run unchanged apart from the model on the same 40, after the one change each tool needed to reach the new model (Codex CLI from 0.158.0 to 0.159.2, pi-review onto pi 0.99.1, a custom model entry for OpenCode). Share of comments graded correct, then share of the 115 defects found: pi-review’s panel 78% to 91% and 68% to 70% (78 and 80 defects); open-code-review 76% to 88% and 68% to 66% (78 and 75.5); OpenCode 66% to 89% and 60% to 50% (69.5 and 57.5); Codex CLI 78% to 94% and 56% to 56% (64 and 64.5); PR-Agent 61% to 63% and 30% to 34% (34.5 and 39.5). Unrounded, the five moved by +1.7, −2.2, −10.4, +0.4 and +4.3 points. On GPT-6.1 Sol the six reviewers of figure 6 run from 39.5 to 98 of 115, 34.3% to 85.2%, 50.9 points apart. OpenCode’s other variants were not re-run on GPT-6.1 Sol, so a different OpenCode configuration may now be its best.
• The single-setting rows. Figure 4 plots the paper’s table for the 40, every row on GPT-6 Sol, as rounded there, except QuickReview’s row, printed to one decimal, as defects found of 115, share and cost per pull request. Unrounded, Codex CLI’s move to high effort is 11.3 points, its gain from the PR text 10.0, OpenCode’s 10.9 and pi-review’s from medium to high 10.0. The body’s 6 cents and 2.8 times also use unrounded costs, as the papers do: Codex CLI rose from $0.187 to $0.244 a pull request, and pi-review’s panel cost $0.465 against $0.169 for one reviewer. QuickReview low 77.5 (67%) at $0.93 and default 98.5 (86%) at $2.11; Codex CLI medium 51 (44%) at $0.19, high 64 (56%) at $0.24, medium with PR text 62.5 (54%) at $0.18; OpenCode through our bridge 49 (43%) at $0.96, native 57 (50%) at $0.40, native with PR text 69.5 (60%) at $0.40; pi-review medium 59.5 (52%) at $0.11, high 71 (62%) at $0.17, high with PR text 72.5 (63%) at $0.18, three-reviewer panel 78 (68%) at $0.46; open-code-review medium 76 (66%) at $0.86, high 78 (68%) at $1.12, medium with PR text 78 (68%) at $0.87.

6. Martian and Qodo, and each benchmark’s own score

• Martian. Martian’s Code Review Bench, offline set: 50 pull requests and 173 answer-key comments, 158 of them in its default scoring profile, which Martian calls Core. The four reviewers ran on GPT-6.1 Sol in their AACR-Bench configurations and were judged by the same GPT-6.1 Sol judge through Martian’s own steps. Precision / recall / F1 / F2 on that profile (F1 weighs precision and recall equally, F2 weighs recall more heavily), issues extracted per PR, cost and mean minutes per PR: QuickReview 16.2 / 65.8 / 26.1 / 40.9, 12.8, $2.84, 26.1; open-code-review 26.7 / 58.9 / 36.8 / 47.4, 7.0, $0.86, 3.3; PR-Agent 41.9 / 36.1 / 38.8 / 37.1, 2.7, $0.09, 1.4; OpenCode 32.7 / 33.5 / 33.1 / 33.4, 3.2, $0.19, 2.4. Martian’s public leaderboard is scored by other judges, so these rows compare only with each other.
• The answer-key audit. One investigator per comment, Claude Opus 5.5 agents reading the code and never any reviewer’s output, with a second, independent investigator for every verdict other than real, found 86 real, 77 not real and 10 disputed. No human has checked these verdicts. Of the 86 real ones, on GPT-6.1 Sol, QuickReview found 75, open-code-review 67, OpenCode 48 and PR-Agent 46. Before its last tuning round, which used 20 of these 50 pull requests, QuickReview on GPT-6.1 Sol found 69 of the 86, two more than open-code-review; on the 30 that round did not use, the frozen build found 43 of 52 against 40 before it.
• Qodo. Qodo’s PR-Review-Bench is 100 pull requests from 8 repositories into which an LLM injected 580 issues, 309 functional bugs and 271 violations of repository rules. Qodo publishes no judge or scorer, so every tool was judged by one GPT-6.1 Sol judge at low reasoning under the matching rule Qodo’s description states, which the paper applies as a comment on the issue’s file within five lines of it. For the four tools, all on GPT-6.1 Sol, comments per PR, precision, recall, F1, functional recall, rule recall, cost and mean minutes per PR: QuickReview 7.1, 37.3, 45.3, 40.9, 52.1, 37.6, $2.48, 19.6; OpenCode 2.6, 64.9, 29.0, 40.0, 43.7, 12.2, $0.20, 2.3; open-code-review 3.5, 47.1, 28.1, 35.2, 45.6, 8.1, $0.46, 1.9; PR-Agent 3.4, 47.2, 27.2, 34.5, 43.7, 8.5, $0.10, 1.2. Qodo built and injected the benchmark, 47% of its issues are rule compliance, and a real defect missing from the issue list counts as a false positive. PR-Agent is Qodo’s own open-source reviewer.
• Rule violations. The frozen build checks every rule the repository writes down and plans a lint-conformance pass when CI runs a linter. Both arrived after the build of 2 October, in the round that mined 59 Qodo issues a vendor’s review found and that build missed. The 2 October build, also on GPT-6.1 Sol, found 14.8% of the rule violations and 50.8% of the functional bugs, so most of the frozen build’s lead on rules came from that round, while its lead on functional bugs was there before it.
• Another judge. For the data release the DeepSeek V4.1 Flash judge of note 2, here at low reasoning, re-scored both benchmarks. Under it the four keep their places. QuickReview has the highest recall and the lowest precision on both (66.5 and 16.9 on Martian, 46.9 and 38.5 on Qodo), and on Martian’s F2 it trails open-code-review, 41.9 against 50.4. No human has checked this judge.
• Noise. These differences come from one run per tool, with no interval computed, apart from the 31-PR Qodo check in note 9.

7. The 20 hard Martian pull requests

• The earlier post. What Claude Code’s multi-agent code review costs and finds measured these 20 pull requests in October, before QuickReview’s rename, so it calls QuickReview Quickchat Review.
• Why these 20. They were chosen before any Claude Code run, as the 20 of Martian’s 50 where QuickReview on GPT-6.1 Sol and DeepSeek V4.1 Flash had done worst. Regression to the mean makes every other reviewer expected to look better there, Claude Code included, but shared failure modes and build changes leave the size of that effect unknown. The earlier post expects the selection to work against QuickReview by an amount it cannot measure. The set stops at 20 of the 50 because Claude Code’s multi-agent max, which the earlier post also ran on these 20, cost about $21 and took about an hour for each finished review.
• What tilts these 20 each way. Claude Code ran at high, which its documentation says broadens coverage. An audit by Claude agents judged 77 of Martian’s 173 answer-key comments not to be real defects the pull request introduces (note 6), and a real problem missing from the key counts against the reviewer that says more. In the other direction, 7 of the 20 are among the 20 Martian pull requests QuickReview’s tuning used, and Claude Code hit the ten-finding cap its prompt sets on 12 of its 20 Opus 5.5 reviews. On Sonnet 5.5, QuickReview skipped every planned pass on 2 of the 20 (below), a failure of its own.
• The runs. Claude Code 2.1.303 /code-review high is one agent running eight review angles in its own context, then deduplication, with no verifier stage; it read only <base>...<head>. QuickReview read the PR’s title and description and the source of imported packages, and ran six code-quality passes on GPT-6 Luna for under 2% of its cost. Each configuration reviewed each pull request once.
• Full results. Serious answer-key issues matched of 27, all matched of 65, unmatched claims per PR, cost per PR and F2: Claude Code on Opus 5.5 22, 43, 9.8, $0.33, 43.2; QuickReview on Opus 5.5 22, 44, 16.2, $4.61, 35.0; Claude Code on Sonnet 5.5 21, 47, 11.9, $0.11, 43.1; QuickReview on Sonnet 5.5 20, 37, 6.2, $1.17, 43.9.
• Scoring. Martian’s own pipeline at a pinned commit, with one GPT-6.1 Sol judge at low reasoning for every configuration. Serious issues are the answer-key comments Martian labels Critical or High, and an unmatched claim is one that matches no answer-key comment. About ±2 answer-key issues per configuration is treated as noise, a prior from earlier repeated scoring; no variance was measured on these runs.
• QuickReview’s failures. On Sonnet 5.5 its session ran none of its planned passes and no validator on 2 of the 20 pull requests (“I skipped the lens passes and the validator and reviewed the diff myself”; one went out as an approval) and launched fewer passes than planned on 3 more. On Opus 5.5 every planned pass and the validator ran on every pull request.
• Claude Code’s multi-agent review. /code-review max with every subagent forced to Sonnet 5.5 matched 23 of 27 serious and 47 of 65 issues, against 21 and 47 for one Sonnet 5.5 agent, at a recorded $398.03 against $2.12 for the 20. Five of its 19 finished reviews launched no verifier.
• The pairwise judges. QuickReview on Opus 5.5 rated 110 Elo points above Claude Code high on Opus 5.5 (95% interval +31 to +196), 29 wins to 11 over both judges; the GPT-6.1 Sol judge alone gave +52 (−68 to +182) and the Opus 5.5 judge +217 (+99 to +377). The pairwise judge was added after the answer-key scores were in, and over all their verdicts the two judges picked the same winner 56% of the time.
• All 50. Claude Code high on Opus 5.5 matched 51 of the 66 serious answer-key issues at $0.36 a pull request, with an F2 of 44.9, and QuickReview on Sonnet 5.5 matched 49 at $1.32, with an F2 of 47.4. No same-model pair ran on all 50.

8. Costs, times and the arithmetic

• Prices. Every cost is at list prices, not a bill. The OpenAI-model runs went through a ChatGPT subscription login, so their costs are list-price computations from token counts; Claude Code’s are its own reported cost, an Anthropic list-price equivalent.
• Ratios. $3.24 / $0.16 = 20.3, $2.06 / $0.14 = 14.7 and, for the GPT-6 Sol pair, $1.71 / $0.15 = 11.4; the GPT-6.1 Sol totals, $380 against $27, give 14.1. On the Martian 20, $4.61 / $0.33 = 14.0 and $1.17 / $0.11 = 10.6, or 11.0 from the totals, $23.35 against $2.12.
• The 145 pull requests of the headline. Computed from the published dataset’s per-review costs and comment counts and its per-defect verdicts, for the 145 only. Total cost, mean per pull request, comments and median minutes: QuickReview’s 1 October build $204.10, $1.41, 807 (5.6 a pull request), 9.5; its frozen build $309.40, $2.13, 1,047 (7.2), 16.7; its low preset $131.90, $0.91; pi-review’s panel $41.81, $0.29, 468 (3.2), 1.6; open-code-review $85.85, $0.59, 484, 2.0; Codex CLI $21.78, $0.15, 279 (1.9), 1.9; OpenCode $25.26, $0.17, 281, 2.2; PR-Agent $10.37, $0.07, 319, 1.8; Claude Code on Opus 5.5 $29.98, $0.21. Over all 185 the means are $1.36 for the 1 October build, $2.06 for the frozen build, $0.88 for the low preset, $0.27, $0.53, $0.14, $0.17 and $0.07 for the five third-party tools in the same order, and $0.20 for Claude Code.
• Marginal cost on the 145. Against Codex CLI, $204.10 − $21.78 = $182.32 bought 321 − 226 = 95 more defects (2 high, 15.5 medium, 77.5 low), $1.92 each, and 17.5 more high- or medium-severity ones, $10.42 each, with 807 − 279 = 528 more comments, about 30 for each of the 17.5. Against pi-review’s panel, $204.10 − $41.81 = $162.29 bought 321 − 270 = 51 more (4 high, 2 medium, 45 low), $3.18 each, and 6 more high- or medium-severity ones, $27.05 each, with 339 more comments. The cost ratios are 9.4 and 4.9. No interval was computed for these.
• Comment grades on the 145. Of QuickReview’s 807 comments the judge graded 61.1% correct and worth raising before merge, 33.5% correct and minor, 2.5% partly correct and 2.7% incorrect; of Codex CLI’s 279, 90.3%, 2.2%, 3.2% and 4.3%.
• Marginal cost of the frozen build, all 185. On GPT-6.1 Sol, $380 − $27 = $353 bought 461 − 287 = 174 more defects, $2.03 each; 87.5 − 81 = 6.5 more high-severity ones, $54 each; and 222 − 182.5 = 39.5 more high- or medium-severity ones, $8.94 each. On Opus 5.5 over the 40, 40 × ($3.24 − $0.16) = $123.20 bought 119.5 − 75 = 44.5 more, $2.77 each; 32 − 25.5 = 6.5 more high, $18.95 each; and 62.5 − 47.5 = 15 more high or medium, $8.21 each.
• How firm the high-severity price is. It rests on 6.5 extra defects on each model. On GPT-6.1 Sol the 95% interval on that lead, 0.6 to 14.1 points of the 90, spans about $28 to $650 an extra high-severity defect, and the interval on the high- and medium-severity lead, 11.3 to 22.4 points of the 235, spans about $6.70 to $13.30. The Opus pair’s figures have no interval.
• Reading load. On GPT-6.1 Sol the frozen build posted 6.9 comments a pull request, or about 1,280 over the 185, against Codex CLI’s 343, so roughly 930 more, or about 24 for each of the 39.5 extra high- or medium-severity defects. At 100 pull requests a day the price gap is (2.06 − 0.14) × 100 = $192 a day at list prices.
• Time. Minutes are wall-clock medians under a shared scheduler with contention, not a speed comparison.

9. Tuning, and the pull requests kept out of it

• Why the headline uses the 145. They were set aside mechanically after QuickReview’s last benchmark-driven change of September and never looked at for its design until the 1 October build had run on them. The 40 pull requests the third-party configurations were picked on are part of the 145, which favours those tools there.
• No pull request is untouched by every round. QuickReview’s September changes were designed from its misses on its 40 tuning pull requests, the same 40 as the Opus pair, the 20 kept out of the last round included; the other 145 were held out from every round up to the build of 1 October. The frozen build’s changes were designed from the compared runs’ misses on those 145, on 20 of Martian’s pull requests and on 59 Qodo issues a vendor found and the 2 October build missed, and validated on 20 of the 40.
• What changed. Between the 1 October build and the frozen build of 4 October, both on GPT-6.1 Sol, QuickReview gained a pass that compares base and head behaviour on modified or deleted lines where base behaviour is at risk, a pass planned only when CI runs a linter or formatter over a changed file, shared instructions such as checking a confirmed mechanism’s sibling sites, a per-file coverage check that sends a targeted sweep to changed source files no pass commented on, a deterministic merge of duplicate findings into one comment per root cause, and high reasoning for three passes. On all 185 pull requests the frozen build found 10.0 points more, with a 95% interval of +6.8 to +13.3.
• What was kept out. On the other 20 of the 40, the frozen build on GPT-6.1 Sol found 45.5 of 55 defects (83%), against 39.5 (72%) for pi-review’s panel, 38.5 (70%) for the 1 October build and 29 (53%) for Codex CLI, all on GPT-6.1 Sol, and 36 (65%) for Claude Code on Opus 5.5. On the 145, before the frozen build’s round looked at them, the 1 October build found 321 of 394 (81%) with its default and 284.5 (72%) with its low preset, against 270 (69%) for pi-review’s panel, 242 (61%) for open-code-review, 226 (57%) for Codex CLI, 203.5 (52%) for OpenCode and 150 (38%) for PR-Agent, all on GPT-6.1 Sol, and 249.5 (63%) for Claude Code on Opus 5.5. Of the 56 high-severity defects there they found 54, 52 (low preset), 50, 41.5, 52, 48, 35.5 and 47, and of the 224 low-severity ones the six of the headline found 171, 126, 117.5, 93.5, 77.5 and 60. After its round the frozen build found 356 of the 394 (90%). On Martian’s 30 held-out pull requests the frozen build found 43 of 52 real answer-key comments, and on the 31 Qodo pull requests no development used, its recall rose 7.8 points over the build of 2 October (95% interval +2.1 to +14.1).

10. Code review research

• Peter C. Rigby and Christian Bird, “Convergent Contemporary Software Peer Review Practices”, ESEC/FSE 2013. They studied Android and Chromium OS, Bing, Office and Microsoft SQL Server, and projects at AMD, against a traditional inspection at Lucent and six open source projects. The quotation is verbatim from their discussion of the number of reviewers, which also says reviews are expensive “because they require reviewers to read, understand, and critique an artifact” and cites Porter et al. finding “that two reviewers discovered as many defects as four reviewers”. The application to AI review passes is mine.

11. Noise, intervals and one run

• Intervals. 95% intervals are paired bootstraps over pull requests, 10,000 resamples, and cover pull-request sampling only, because each reviewer ran and was judged once. On GPT-6.1 Sol the frozen build’s lead over Codex CLI is +7.2 points on high-severity defects [+0.6, +14.1], +16.8 on high and medium together [+11.3, +22.4] and 33.5 points on all 520. The 1 October build’s high-severity lead over Codex CLI, +2.8 [−3.0, +8.7], is not established; with 90 high-severity defects, gaps of a few points are below what this benchmark can resolve. No interval was computed for the Opus pair.
• Run-to-run spread. It was measured for QuickReview only. A full re-run of the GPT-6 Sol default found 436 defects against 437.5. Two runs of one earlier build, also on GPT-6 Sol, differed by 1.5 to 2 defects, 4 to 8 points, on single 10-PR sets, so single-digit differences on 40 pull requests are not established. The configuration picks’ move to a new model is in note 5.

12. Conflicts of interest

• Quickchat AI builds QuickReview, which competes with every other reviewer here; I built it, designed these tests, ran them and wrote this post. The same assistant that designed QuickReview’s benchmark levers orchestrated the AACR-Bench key and wrote the judge prompt. Quickchat AI runs on OpenAI and Google Cloud.
• The AACR-Bench judge shares a model family with the seven OpenAI-based reviewers, QuickReview’s GPT rows included, and the key builder and the second judge share a family with Claude Code. Vendor-run evaluations are known to favour their sponsor, and none of these checks substitutes for an independent replication.
• The lead is largest where an alignment between QuickReview’s design and the key’s definition of a defect would show first, at low severity and on maintainability defects, where the 1 October build on GPT-6.1 Sol found 81% of 72 against Codex CLI’s 16%; without those 72 it found 78% of the other 448 against 71% for pi-review’s panel.

13. Data

• The dataset. Mateusz Jakubczak, Real defects in code review: QuickReview benchmark data, Quickchat AI, 2026, huggingface.co/datasets/quickchatai/quickreview-bench, doi:10.57967/hf/11598, with its code at the tag bench-data-v1. It has been public since 12 November 2026 under the Apache-2.0 licence and needs no login. It holds the answer key, every review, every judge verdict with the judge’s reasoning, each review’s cost, time and comment count, and, in its second_judges folder, the DeepSeek judge’s verdicts.
• The repository. At that tag it holds the export, the scorers, both papers linked above and a claims check that recomputes every benchmark number the papers and the repository’s benchmark page print, from the dataset and without a model call. The answer key is there too, each of the 520 defects with its quoted evidence and the sources that raised it.
• Counts made for this post. The benchmark paper prints how many defects each reviewer found on the 145 pull requests, the 38% to 81% of the headline among them. Every other figure on the 145 is my own count from the dataset’s tables: the split by severity, the 250 defects that no reviewer’s comment put in the key, the cost, comments, minutes and comment grades of each reviewer, and the price of each extra defect. The 332 of 369 in note 2 and the third judge’s figures on the 145 are counted the same way. They use the pull requests the dataset labels rest145 and, for the primary judge, the verdicts labelled gpt-6-sol-high, with a partly found defect as half. The paper’s own tables cover all 185, and the dataset reproduces them.

14. The example pull request

• The pull request is number 831 of microsoft/typescript-go, one of the 145. The defect is graded high: the synthetic import declaration shares its bindings and module specifier with the JSDoc tag, and for any host that is not a function declaration, binding the host afterwards re-parents those shared nodes, so the checker reaches a node kind it does not handle and panics. The pull request’s conformance tests use only function hosts, which is why they pass.
• On GPT-6.1 Sol, QuickReview’s 1 October build at its default and pi-review’s panel found it, open-code-review found it partly, and Codex CLI, OpenCode, PR-Agent and QuickReview’s low preset missed it. Claude Code on Opus 5.5 missed it too. The benchmark paper quotes the code and each reviewer’s comment.