---
title: "My AI code reviewer kept missing bugs in the very lines it commented on"
pubDate: 2026-11-13
description: "On 185 pull requests, QuickReview's September build missed 80 bugs outright. For 49 of them it had left a comment inside the bug's own lines, about something else. Chance gives about 37."
author:
name: "Mateusz Jakubczak"
image: "/blog-assets/authors/mateusz_jakubczak.jpeg"
image:
url: "/blog-assets/posts/my-ai-code-reviewer-kept-missing-bugs-in-the-lines-it-commented-on_bg_light.png"
alt: "On a cream background, a white card titled \"QuickReview\", badged \"GPT-6 Sol\" and labelled \"posted comment\", links by a solid amber line to the first of three lines inside one red outline in a \"Files changed\" diff. That line is highlighted amber and carries a comment-bubble chip reading \"about something else\". A card titled \"Answer key\", labelled \"verified bug\" with a red bug icon, links by a dashed line to the last line inside the same red outline, marked \"missed\"."
tags: ["engineering"]
minutesToRead: 9
faq:
- question: "Where do AI code reviewers miss bugs?"
answer: "Often in lines they have already commented on. On 185 pull requests, QuickReview's September build on GPT-6 Sol missed 80 bugs outright. For 49 of them, one of its comments started inside the bug's own lines and was about something else. Chance gives about 37. Six other reviewers show the same excess over chance. They comment less, so most of their misses are in code they never mentioned."
- question: "Does a comment from an AI reviewer mean the code under it was checked?"
answer: "No. Take the bugs whose lines QuickReview's September build had already commented on about a different bug. It missed 20% of them, against 15% of all bugs. For its later build the two rates are 12% and 11%. A comment shows the reviewer was there. It does not show the lines are clean."
- question: "Do other AI code reviewers miss bugs under their own comments too?"
answer: "Yes. On the same 185 pull requests, each of six other reviewers missed 33 to 66 bugs that had one of its own comments inside the bug's lines. For all six that is more than chance gives on average, and for five it is clear of the chance range. The one inside its range is Claude Code, run here as the local /code-review command of version 2.1.283 on Opus 5.5 at medium effort."
- question: "How should I read AI code review comments?"
answer: "Treat each comment as a pointer to risky code. Read the whole block under it and look for a second bug. Follow any consequence the comment names one step further. Compare the code with what the pull request says it does. A re-run of the same reviewer found 27 of the 80 bugs its first run missed, so asking again helps too."
- question: "Would a looser filter catch the bugs an AI reviewer misses?"
answer: "Few of them, on this reviewer. QuickReview's September run dropped 271 findings before posting. By one model's labels, those findings stated 5 of the 80 missed bugs. More reasoning did more. With the same pass plans, the default preset found 103 of the 170 bugs the low preset missed."
- question: "Which models and versions were measured?"
answer: "QuickReview's September build and its low preset on GPT-6 Sol, and its build frozen on 4 October 2026 on GPT-6.1 Sol; pi-review 0.18.0, open-code-review 1.3.1, Codex CLI 0.159.2, OpenCode 1.18.33 and PR-Agent 0.46.0 on GPT-6.1 Sol; Claude Code 2.1.283's local /code-review on Opus 5.5 at medium. GPT-6 Sol judged every review in this post, and Opus 5.5 judged the September run as well."
- question: "How reliable is this analysis?"
answer: "It rests on one run per reviewer plus one re-run, one LLM judge for most results, and an answer key that Claude agents built partly from QuickReview's own earlier comments. No human has checked the answer key or the judge yet. A second judge marks the same 80 misses and the same 49 on the September run. The result also holds on the bugs that would be in the answer key without any reviewer's comments, 32 of 51. The test records where comments sat. The counts beyond the papers' are my own recounts from the public dataset."
---
## TL;DR
**When my AI code reviewer missed a bug, it had often already commented on those very lines about something else, more often than chance explains. A comment tells you the reviewer was there and nothing about what else is in those lines, so look for a second bug under every comment.**
Ollama's pull request #10135 capped its progress display at the height of the terminal. The description says the display "only outputs the last lines that fit into the terminal". The code does this:
```go
for i := range min(len(p.states), termHeight-1) {
fmt.Fprint(p.w, p.states[i].String(), "\033[K")
```
That loop prints the first lines that fit, which are the oldest. Once a model has more files than the terminal has rows, the user watches finished spinners, and the uploads still running are never drawn.
My reviewer left a comment on that exact line. It was marked a blocker, at confidence 0.99, and it was about cursor movement: "Earlier terminal output gets overwritten or cleared when the progress list outgrows the terminal." The benchmark's judge graded that comment incorrect. The bug in the line went unmentioned. Four of the six other reviewers I compare against found it[10](#n10).
I build QuickReview, an open-source AI reviewer for GitHub pull requests (PRs)[1](#n1). A benchmark score tells me how many bugs it missed and nothing about where they were. I wanted to know whether its misses sit in code it never mentioned or in code it had already commented on. The Ollama miss turned out to be the common case: a missed bug with one of the reviewer's own comments sitting on it.
## How I tested it, and three caveats
AACR-Bench has 185 runnable PRs from public repositories. I score each review against an answer key, a list of the known bugs in those PRs. It marks 520 bugs and, for each bug, the line ranges it lives in, typically five lines long[2](#n2). The answer key, every review and every judge verdict are public in a [dataset on Hugging Face](https://huggingface.co/datasets/quickchatai/quickreview-bench) (DOI [10.57967/hf/11598](https://doi.org/10.57967/hf/11598)), so the counts in this post can be recomputed[15](#n15).
QuickReview's September build, on GPT-6 Sol, missed 80 of the 520 outright, with no credit from the judge[3](#n3). For each of the 80 I checked one thing: whether one of the review's own posted comments started inside the lines the answer key marks for that bug. When one did, I say the bug sat under a comment.
Three caveats before the result:
- **The answer key and the judge are machines.** Claude agents built the answer key, partly from this reviewer's own earlier comments, and an LLM judge decides what counts as found. No human has checked either[2](#n2).
- **QuickReview comments a lot.** It posted 6.4 comments a PR in this run. A reviewer that busy lands inside many bugs' lines by accident, so every count below comes with what chance gives[4](#n4).
- **The September build is older than the published one.** I ran the same test on the build behind QuickReview's published numbers, frozen on 4 October. I call it the frozen build, and it is in the first chart[7](#n7).
## 49 of the 80 missed bugs had a comment inside their own lines
Of the 80 missed bugs, 49 sat under a comment. To see what chance gives, I moved the same 80 bugs to random changed lines of the same files and counted again. On average about 37 then had a comment inside them, and 95% of the random draws gave 30 to 43.
*In every QuickReview run I tested, more missed bugs had one of its comments inside their own lines than random placement gives. For the September build it was 49 of 80 against about 37.*
The result holds where it could have failed:
- **On PRs the build was never tuned on.** The September build was designed from its misses on 40 of the 185 PRs. On the other 145 it missed 58 bugs, and 31 of them sat under a comment, against about 24 by chance[4](#n4).
- **On a re-run.** Two days later the same build missed 80 bugs again, 51 of them the same ones. This time 41 of the 80 sat under a comment, against about 31 by chance[9](#n9).
- **On the frozen build.** The frozen build, on GPT-6.1 Sol, missed 55 bugs, and 27 of them sat under its own comments, against about 21 by chance. That is above chance by a thin margin, since the random draws reach 26. It is also no longer most of the misses. None of that build's comments went into the answer key[7](#n7).
- **With a second judge and a cleaner answer key.** Opus 5.5, given the same prompt, marks the same 80 bugs missed and the same 49 under a comment. The answer key has 369 bugs that would be in it even if no reviewer's comment had been a candidate. On those 369, the count is 32 of 51 misses[3](#n3).
Chance here means random changed lines. Real bugs are not random. They sit in the hard part of a change, and that is where a reviewer comments too. So this result says where the misses are. It does not say a comment caused a miss.
The misses were mostly small. By the answer key's severities, 35 of the 49 are low, 12 medium and 2 high[5](#n5).
## The comment on top was usually right, and about a different bug
In all 49 cases the comment was about something other than the bug under it. I sorted the 49 by what the comment was about[5](#n5):
- **26 sat under a comment about another real bug,** one the answer key lists and the review found. In 17 of them the bug it found was the more severe of the two.
- **19 sat under a comment that was true and minor,** about nothing in the answer key.
- **4 sat under a comment that was wrong.** The Ollama line is one of these.
Two of the 26 show what that looks like[10](#n10). In each, the review found one real bug and missed a second one in the same lines.
**Elasticsearch #118998** makes the `content` field of a chat message optional, so an assistant message that carries only tool calls is accepted. QuickReview left a blocker on those lines: a message without content parses, then throws a null-pointer exception when the request is serialized between nodes. That bug is real, and the answer key rates it low.
The same lines hold a second bug, which the answer key rates medium. The parser accepts a missing `content` and still rejects an explicit `"content": null` with a 400, and null is what OpenAI-style clients send. So the PR did half of what it set out to do, and the review found the smaller problem with that half. Of the six other reviewers, only Claude Code found the bigger one.
**three.js #30750** welds points that sit closer than 0.0002 units before it triangulates a shape. QuickReview's comment on that block was a question: the cutoff "can erase a valid contour: a triangle with sides around `0.0001` has each successive vertex spliced away, eventually leaving no contour for the caps or sidewalls." That bug is also real, and it is in the answer key.
The missed bug is one step further along the same lines. When the ring that vanishes is a hole, the constructor throws a `TypeError` inside the triangulator, on input that built fine before the change. The review saw the points disappear and did not follow them to the crash. Of the six other reviewers, only PR-Agent did.
## A comment about one bug did not protect the next
Suppose a reviewer finds one bug in a block and then checks the rest of the block. Bugs under its comments would then be missed less often than bugs elsewhere. They were not.
For 128 bugs in the answer key, the September build had already commented on their lines about a different bug. It missed 26 of those 128, which is 20%. Over all 520 bugs it missed 15%. For the frozen build the two rates are 12% and 11%.
I checked nine runs this way. The two rates are never more than 8 points apart, and in seven of the nine the bugs under a comment were missed slightly more often[8](#n8).
So a comment on a block tells you nothing about the rest of the block. The reviewer had found one bug there and written it up, and its review said nothing about the second.
## Six other reviewers show the same excess
I ran the same test on the six other reviewers in the benchmark paper's final comparison[7](#n7).
*Each of six other reviewers missed more bugs under its own comments than random placement gives on average. Five are clear of the chance range, and Claude Code's 33 is inside its range of 17 to 34.*
These reviewers post 1.9 to 3.2 comments a PR, against QuickReview's 6.4. So the bugs under their comments are a smaller share of what they missed. Codex CLI missed 232 bugs, and 43 were under its own comments. Most of what these tools miss is in code they never mentioned.
The excess over chance is still there for all six. For five of them it is outside the range random placement produces.
Claude Code here is version 2.1.283's local `/code-review` command on Opus 5.5 at its default effort, medium. It is not Anthropic's managed Code Review service[7](#n7).
## The filter did not throw these bugs away
QuickReview reviews a PR in separate passes, each looking for one kind of problem, and filters what the passes find before it posts. My first guess was that the missed bugs had been found and then thrown away.
They had not. The September run dropped 271 findings before posting. GPT-6.1 Sol read the dropped findings that sat near a missed bug and labelled what each one said. By those labels, the dropped findings stated 5 of the 80 missed bugs[6](#n6). So the filter accounts for few of the misses.
There is one explanation I could not rule out. The session also merged 1,456 findings into other comments as duplicates, and nobody labelled those. If a merge folded a second bug into a comment about the first and lost it there, the result would look like the Elasticsearch case from outside[6](#n6).
Two things did recover missed bugs: more reasoning and a second run.
A pass plan is the list of passes QuickReview chooses for a PR. The low preset runs the same pass plans as the default preset, with every model call at low reasoning effort. The default found 103 of the 170 bugs the low preset missed. Of those 170, 68 sat under the low preset's own comments, and the default found 35 of the 68.
The re-run found 27 of the first run's 80 misses with no change in effort at all[9](#n9).
## What to do with this
If you read AI review comments:
1. **Treat a comment as a pointer to risky code.** In this data, 49 of 80 misses were in the code under a comment, in blocks a few lines long. Read the whole block and look for a second bug.
2. **Follow the consequence one step further.** If the comment says a value can end up empty, ask what the next line does with an empty value.
3. **Check the code against what the PR says it does.** The Ollama bug and the Elasticsearch bug are both a gap between the description and the lines under the comment.
4. **Ask again.** A second run of the same reviewer found a third of what the first one missed.
A controlled study of 29 experts by Tufano and colleagues looked at people who start from an automated review. It found that they "tend to focus on the code locations indicated by the LLM rather than searching for additional issues in other parts of the code"[11](#n11). On this data that is a good place to look, as long as you read the code and not only the comment.
If you build a reviewer, send it back to the blocks it has already commented on, and raise its reasoning before you loosen its filter.
QuickReview's frozen build added an instruction to check a confirmed bug's sibling sites and consequences. It found all three bugs above. On three.js it then missed the cutoff bug the September build had found, in the same lines[12](#n12).
## Limits
Each reviewer ran once, apart from the one re-run. One LLM judge scored every review, and a second judge checked the September run. The data release adds a third judge on the frozen build and the six other tools. On its verdicts each count of misses under a comment moves by at most one. Claude agents built the answer key, and no human has checked the answer key or the judge[2](#n2).
The test records where comments sat. It cannot show what a model read or why it stopped.
The public QuickReview report counts something looser: a comment within five lines of the bug. That gives 56 of the 80. I dropped that test for this post because random placement gives 49 to 52[4](#n4).
Every count here that goes beyond the papers is my own recount from the public dataset. The dataset's claims check covers the papers' numbers and does not cover my recounts[15](#n15).
QuickReview was tuned on this benchmark. It is built to find as many bugs as it can, and it pays for that. The frozen build found 461 of the 520 bugs, 89%. It cost $2.06 a PR and posted 6.9 comments a PR. That cost is 4 to 31 times the cost of the six other tools[13](#n13) [14](#n14). [The next post in this series](https://quickchat.ai/post/same-model-six-code-reviewers-38-to-81-percent-of-the-bugs) works out what each extra bug cost.
## Frequently asked questions
### Where do AI code reviewers miss bugs?
Often in lines they have already commented on. On 185 pull requests, QuickReview's September build on GPT-6 Sol missed 80 bugs outright. For 49 of them, one of its comments started inside the bug's own lines and was about something else. Chance gives about 37. Six other reviewers show the same excess over chance. They comment less, so most of their misses are in code they never mentioned.
### Does a comment from an AI reviewer mean the code under it was checked?
No. Take the bugs whose lines QuickReview's September build had already commented on about a different bug. It missed 20% of them, against 15% of all bugs. For its later build the two rates are 12% and 11%. A comment shows the reviewer was there. It does not show the lines are clean.
### Do other AI code reviewers miss bugs under their own comments too?
Yes. On the same 185 pull requests, each of six other reviewers missed 33 to 66 bugs that had one of its own comments inside the bug's lines. For all six that is more than chance gives on average, and for five it is clear of the chance range. The one inside its range is Claude Code, run here as the local /code-review command of version 2.1.283 on Opus 5.5 at medium effort.
### How should I read AI code review comments?
Treat each comment as a pointer to risky code. Read the whole block under it and look for a second bug. Follow any consequence the comment names one step further. Compare the code with what the pull request says it does. A re-run of the same reviewer found 27 of the 80 bugs its first run missed, so asking again helps too.
### Would a looser filter catch the bugs an AI reviewer misses?
Few of them, on this reviewer. QuickReview's September run dropped 271 findings before posting. By one model's labels, those findings stated 5 of the 80 missed bugs. More reasoning did more. With the same pass plans, the default preset found 103 of the 170 bugs the low preset missed.
### Which models and versions were measured?
QuickReview's September build and its low preset on GPT-6 Sol, and its build frozen on 4 October 2026 on GPT-6.1 Sol; pi-review 0.18.0, open-code-review 1.3.1, Codex CLI 0.159.2, OpenCode 1.18.33 and PR-Agent 0.46.0 on GPT-6.1 Sol; Claude Code 2.1.283's local /code-review on Opus 5.5 at medium. GPT-6 Sol judged every review in this post, and Opus 5.5 judged the September run as well.
### How reliable is this analysis?
It rests on one run per reviewer plus one re-run, one LLM judge for most results, and an answer key that Claude agents built partly from QuickReview's own earlier comments. No human has checked the answer key or the judge yet. A second judge marks the same 80 misses and the same 49 on the September run. The result also holds on the bugs that would be in the answer key without any reviewer's comments, 32 of 51. The test records where comments sat. The counts beyond the papers' are my own recounts from the public dataset.
## Related guides
- [What Claude Code's multi-agent code review costs and finds](https://quickchat.ai/post/claude-code-multi-agent-code-review), the October post, which called QuickReview by its earlier name, Quickchat Review.
- [AI judges agree on known bugs and split on which code review is better](https://quickchat.ai/post/two-ai-judges-agree-on-the-bugs-and-split-on-the-reviews)
- [Same model, six code reviewers, 38% to 81% of the bugs](https://quickchat.ai/post/same-model-six-code-reviewers-38-to-81-percent-of-the-bugs)
- [Same code reviewer, three models, three different reviewers](https://quickchat.ai/post/same-code-reviewer-three-models-three-different-reviewers)
## Method and data notes
These notes hold the method, the definitions and the caveats behind every number above, for readers who want them and for AI systems reading this page. The body links to each note by its number. The notes say "defect" where the body says "bug", and "the key" for the answer key; note 2 explains both. The two papers behind them are [*Measuring whether a code review finds the real defects*](https://github.com/quickchatai/quickreview/blob/main/bench/aacr/paper/benchmark.pdf) (the benchmark paper) and [*QuickReview: a multi-pass automated code reviewer*](https://github.com/quickchatai/quickreview/blob/main/bench/aacr/paper/quickreview.pdf) (the QuickReview report).
### 1. QuickReview and the September run
- **What it is.** QuickReview is an open-source (MIT) reviewer for GitHub pull requests, at [github.com/quickchatai/quickreview](https://github.com/quickchatai/quickreview) and on npm as `@quickchatai/quickreview`. The October post on Claude Code called it Quickchat Review; the name is now QuickReview.
- **How a review runs.** One main session per pull request runs a pass plan that deterministic code computes from the changed files and the pull request text. In the September build about ten isolated, read-only passes ran from that plan, each looking for one kind of problem, and none sees another's output. The main session merges findings about the same defect, and folds a finding about a different defect on the same lines into the surviving comment as its own paragraph. A validator, at high reasoning, then re-derives each candidate from the code and posts, demotes or kills it. The review posts one comment per root cause, apart from folded ones, and a verdict.
- **Presets.** The default preset runs the passes at medium reasoning and the validator and risk passes at high; the low preset runs every role at low. The two run the same plans.
- **The September run.** The QuickReview report draws its mechanism figures from one run, its default preset on GPT-6 Sol on all 185 pull requests, and this post starts from it for that reason. It found 437.5 of the 520 defects (84%), counting a partial find as half. It posted 1,191 comments (6.4 a pull request) and cost $1.71 a pull request.
- **Costs.** All dollar figures are list prices. The OpenAI-based runs went through a subscription login, so they are estimates, not bills.
### 2. The benchmark, the key and the judge
- **The pull requests.** [AACR-Bench](https://arxiv.org/abs/2601.19494) v1.0 holds 200 pull requests from 50 public repositories, and its harness scores 196. Eleven could not be run because GitHub no longer serves their reviewed head commit, which leaves 185 pull requests in ten languages. Every runnable pull request is used, so there is no selection step.
- **The key.** The answer key holds 520 defects (90 high, 145 medium, 285 low), each a concrete problem the pull request introduces or newly exposes, marked at one or more line ranges. The key marks 1,286 ranges, a median of 2 per defect, each a median of 5 lines long, with a 90th percentile of 19. Claude agents on Opus 5.5 built the key from a blind hunt of the code, AACR-Bench's own reference comments and the comments of earlier reviewer runs. They verified every candidate against the code without seeing who raised it, and recorded 3,093 rejected candidates with their reasons. **No human has checked any part of the key.**
- **Bugs and defects.** The body calls the key's entries bugs. The key calls them defects, and by kind 229 of the 520 are bugs, 72 maintainability problems such as stale documentation, 57 incomplete fixes and 42 build problems, with the rest spread over smaller kinds.
- **The key and the September run.** On 145 of the pull requests the September run's own comments were among the key's candidates, and they were the largest single source. None of the 58 defects the September run missed outright on those 145 pull requests came from one of that run's own comments. The GPT-6.1 Sol runs, the frozen build's included, were never candidates; the key was finished before they ran.
- **The judge.** GPT-6 Sol at high reasoning, the papers' primary judge. It reads the whole review against the key and grades each defect found, partial or missed, and grades each comment correct-valuable, correct-minor, partly correct, incorrect or unverifiable, naming the key defect it finds, if any. On an earlier set of 1,295 reviews, judges from two model families agreed on 98.7% of 3,640 such verdicts. The [first post in this series](https://quickchat.ai/post/two-ai-judges-agree-on-the-bugs-and-split-on-the-reviews) covers how far such a judge can be trusted.
- **Which runs had a second judge.** Opus 5.5 also graded the September run. For the data release, DeepSeek V4.1 Flash re-graded the frozen build's run and the six other tools' runs from the review text alone, without the code. The benchmark paper reports that it gave the same verdict as GPT-6 Sol on 97.9% to 99.6% of each reviewer's defects. Apart from the second-judge checks named in this post, every count here uses GPT-6 Sol's verdicts. On DeepSeek's verdicts, by my recount, each of those seven runs' count of misses with a comment inside their lines moves by at most one, and the frozen build's is 28 of 56.
### 3. Missed outright, and the test
- **Missed outright.** The judge gave the defect no credit. The September run missed 80 outright and partly found 5. The QuickReview report gives both counts: 80 defects missed outright, or 82.5 missed credits when a partial find counts as half.
- **The test.** A missed defect counts when a posted comment's first line falls inside one of the line ranges the key marks for it. For the September run that is 49 of the 80, on 40 pull requests. In 41 of the 49 the range the comment started in was 10 lines or shorter.
- **The second judge.** Given the same prompt, Opus 5.5 marks the same 80 defects missed outright in the September run, and the same 49 of them with a comment inside their lines.
- **The key's own sources.** The key was built partly from reviewers' comments, so some defects may have entered it through the very comments this post measures position against. The benchmark paper's independent-source check keeps the 369 defects that the blind hunt or an AACR reference raised, which would be in the key had no reviewer's comment been a candidate. On that subset the September build missed 51 outright, 32 of them (63%) with a comment inside their lines, against 49 of 80 (61%) on the whole key. The frozen build missed 34, 17 of them inside (50%), against 27 of 55 (49%). A verifier with every reviewer's comments in view still wrote each defect's definition, so the subset is cleaner and not clean.
- **What it measures.** Position only. A comment inside a defect's lines shows where the review spoke and cannot show what the model attended to.
### 4. What chance gives
- **The placement.** I placed each run's missed defects at random changed lines of the same files, 200 times, and counted how many then had a posted comment starting inside them. In the placement the body and the charts use, every marked range moves on its own to a random changed line of its file, keeping its length. Ranges in files outside the diff stay where they are.
- **The counts.** September build: 49 of 80 against a mean of 36.9, with 95% of draws between 30 and 43. On the 145 pull requests never used in its design: 31 of 58 against 24.0 (20 to 29). On the 40 it was designed from: 18 of 22 against 12.8 (9 to 16). The re-run: 41 of 80 against 31.1 (25 to 38). The frozen build: 27 of 55 against 20.7 (16 to 26).
- **The other placement.** Moving a defect's ranges in one file together, keeping their layout, gives lower chance figures: 32.5, 20.8, 11.7, 26.0 and 17.8 for the same five rows. I use the placement that is harder on my claim.
- **The five-line test, and why I dropped it.** The QuickReview report classes a miss by whether a posted comment starts within five lines of the defect, which gives 56 of the September run's 80. A missed defect's ranges plus five lines either side span a median of 37.5 lines, and random placement puts a comment that close for 52.0 or 48.9 of the 80, depending on the placement. On the 145 pull requests never used in the design it is 36 of 58 against 35.4 or 33.0, which is chance. An earlier draft of this post led with that count.
- **What chance does not model.** Defects and comments both concentrate in the complicated part of a change. Random changed lines do not capture that, so the excess shows that misses and comments share locations and does not show that one caused the other.
- **Status.** This is my own recount from the dataset's reviews, verdicts and key and from the pull requests' diffs, which anyone can rebuild from the public base and head commits. The dataset's claims check does not cover it (note 15).
### 5. What the comments on top were, and how serious the misses were
- **Method.** For each of the 49 I took every posted comment starting inside the defect's ranges, with the judge's grade for it and the key defect it named. If any named a different key defect, the miss is counted there (26, and the review had found that other defect in all 26). Otherwise the best grade among them decides: correct-minor (19) or incorrect (4). No such comment named the missed defect itself.
- **Which bug it found.** In the 26, the defect the review found was more severe than the one it missed in 17, equally severe in 6 and less severe in 3.
- **Severity of the misses.** Of the 80, 57 low, 19 medium and 4 high. Of the 49, 35 low, 12 medium and 2 high.
- **The frozen build.** Of its 27, 15 sat under a comment about another key defect, which the review had found in 14, and 12 under a true comment about nothing in the key. 20 are low and 7 medium.
- **The other reviewers.** For the six other tools most such comments were also about another key defect the review had found, 24 of Claude Code's 33 and 37 of Codex CLI's 43 among them.
### 6. The findings the September run dropped
- **Dispositions.** Of the 2,959 candidates in the September run's findings records, 1,456 were merged as duplicates of another finding, 1,034 posted, 164 demoted and posted, 207 killed by the validator, 34 folded, and 64 withheld by the main session. The 271 dropped findings are the 207 and the 64.
- **The labels.** For every defect the run did not fully credit, GPT-6.1 Sol labelled each killed or withheld finding within five lines of the defect as stating it, related to it or about something else. It read the defect's statement and the finding's text, without the code. The labels are one model's and were not checked by hand.
- **The result.** The dropped findings stated 5 of the 80 outright misses. Posting all 271 would add 23% more comments to the 1,191 posted.
- **What the labels leave out.** Merged and folded findings were not labelled, so a merge that carried the missed defect into a comment about another one, and lost it there, is not ruled out.
- **Where to check it.** The dispositions and the labelled figures are in the QuickReview report, and the dataset's claims check recomputes them. The labels and the prompt that produced them are in the repository under `bench/aacr/relabel/`.
### 7. The same test on the frozen build and six other reviewers
- **The runs.** The benchmark paper's final comparison, on the same 185 pull requests, judged by the same judge. Five third-party tools ran in the best configuration among the variants their own documentation ships or recommends, picked by a rule fixed in advance: pi-review 0.18.0's three-reviewer panel (thinking high), open-code-review 1.3.1 (effort high), Codex CLI 0.159.2 (effort high), OpenCode 1.18.33 (native provider, with the pull request title and body) and PR-Agent 0.46.0 (shipped defaults), all on GPT-6.1 Sol.
- **Claude Code.** Claude Code was not put through that selection. It ran its local `/code-review` command, version 2.1.283, on Opus 5.5 at the session default, medium. Anthropic documents that at low and medium effort the review reports only the findings it is most confident in. Its prose answer is parsed into comments with a file and a line, while the judge reads the whole answer, so a finding written without a line can count toward its recall and not toward the position test. The benchmark paper calls its result not a best-shot figure. This is not Anthropic's managed Code Review service or Ultrareview.
- **The frozen build.** QuickReview as frozen on 4 October 2026, on GPT-6.1 Sol; it is the build behind the results in the repository's README and [BENCHMARKS.md](https://github.com/quickchatai/quickreview/blob/main/BENCHMARKS.md). Its comments were never candidates for the key.
- **The counts.** Chance is the mean of 200 random placements, first with every range moved on its own (95% of draws in brackets), then with a defect's ranges moved together.
| Reviewer, model | Found (of 520) | Missed outright | Comment inside the bug's lines | By chance, ranges on their own | By chance, ranges together | Comments per PR |
|---|---|---|---|---|---|---|
| QuickReview frozen build, GPT-6.1 Sol | 461 | 55 | 27 | 20.7 (16 to 26) | 17.8 | 6.9 |
| pi-review panel, GPT-6.1 Sol | 347.5 | 168 | 41 | 24.6 (18 to 32) | 21.2 | 3.2 |
| Claude Code 2.1.283 local `/code-review`, Opus 5.5 | 324.5 | 191 | 33 | 25.0 (17 to 34) | 21.8 | 2.2 |
| open-code-review, GPT-6.1 Sol | 298.5 | 219 | 50 | 38.9 (32 to 47) | 33.4 | 3.1 |
| Codex CLI, GPT-6.1 Sol | 287 | 232 | 43 | 30.9 (24 to 39) | 26.7 | 1.9 |
| OpenCode, GPT-6.1 Sol | 263.5 | 254 | 42 | 30.2 (23 to 40) | 25.9 | 1.9 |
| PR-Agent, GPT-6.1 Sol | 194 | 322 | 66 | 41.1 (32 to 52) | 35.2 | 2.2 |
- **Settings that shape the test.** PR-Agent posts at most 3 key issues and 3 suggestions per chunk, so more of its misses sit where it said nothing by design.
### 8. Whether a comment about one bug protects the next
- **The comparison.** For each run I took the defects that had one of its comments about a different key defect starting inside their lines. The table gives how many there were, the share of them the run missed outright, and the run's overall outright-miss rate.
| Run | Defects under a comment about a different defect | Share of those missed outright | Share of all defects missed outright |
|---|---|---|---|
| QuickReview's September run | 128 | 20.3% | 15.4% |
| QuickReview's re-run | 135 | 19.3% | 15.4% |
| QuickReview's frozen build | 131 | 11.5% | 10.6% |
| Claude Code | 81 | 29.6% | 36.7% |
| pi-review panel | 90 | 35.6% | 32.3% |
| open-code-review | 90 | 38.9% | 42.1% |
| Codex CLI | 82 | 45.1% | 44.6% |
| OpenCode | 77 | 50.6% | 48.8% |
| PR-Agent | 71 | 63.4% | 61.9% |
- **What it says.** These are base rates over different defects, and they do not test what finding one defect does to the chance of finding the next. Two runs, Claude Code's and open-code-review's, missed such a defect less often than their overall rate, and seven more often. The largest gap is Claude Code's 7.1 points.
### 9. Low against default, and the re-run
- **What differs.** The two presets run the same pass plans, 11.0 passes a pull request on average for low against 11.2, and differ in reasoning effort.
- **Defect by defect, on GPT-6 Sol.** The low preset missed 170 defects outright, 68 of them with one of its own comments inside their lines. Of those 68 the default found 35, partly found 2 and missed 31. Of the other 102 it found 68, partly found 3 and missed 31.
- **The re-run.** A full re-run of the September default on 29 September, after three reviewer changes, found 436 defects against 437.5. It also missed 80 outright, 51 of them the same, and found 27 of the first run's 80 misses. Part of every comparison between two runs is this churn; the low preset was not re-run.
### 10. The examples
- **Where they come from.** The September run's 49 misses with a comment inside their lines. I picked three that can be explained in a paragraph. Quotes are verbatim from the posted review. Ollama and three.js are MIT-licensed.
- **ollama #10135 (medium).** The key marks `progress/progress.go` lines 117 to 125, and the comment sat on line 118. Among the reviewers of the final comparison, the pi-review panel, Claude Code, Codex CLI and OpenCode found the defect, and open-code-review and PR-Agent missed it. QuickReview's low preset, its re-run and its frozen build found it.
- **elasticsearch #118998 (medium).** The comment sat on line 126 of `UnifiedCompletionRequest.java`, inside a range the key marks for the defect, and was credited with the pull request's other defect, which the key rates low. Claude Code and QuickReview's frozen build found the missed defect, and the five other tools and the re-run missed it.
- **three.js #30750 (medium).** The comment sat on line 178 of `ExtrudeGeometry.js`, inside the missed defect's range of lines 170 to 194, and was credited with the cutoff defect, also medium. PR-Agent and the frozen build found the crash, and the five other tools and the re-run missed it. The frozen build missed the cutoff defect.
### 11. The study of human reviewers
- **The paper.** Rosalia Tufano, Alberto Martin-Lopez, Ahmad Tayeb, Ozren Dabić, Sonia Haiduc and Gabriele Bavota, ["Deep Learning-based Code Reviews: A Paradigm Shift or a Double-Edged Sword?"](https://arxiv.org/abs/2411.11401), arXiv 2411.11401, a controlled experiment with 29 experts, in the research track of ICSE 2025.
- **What the abstract says.** "Reviewers tend to focus on the code locations indicated by the LLM rather than searching for additional issues in other parts of the code." I read its abstract, and joining their finding to the positions measured here is my own reading.
### 12. What the advice rests on
- **Measured.** Where the misses sat (notes 3 to 5 and 7), the miss rates under a comment (note 8), the dropped findings (note 6) and what more reasoning and a re-run recovered (note 9).
- **Inferred.** The reading habits come from the three examples (note 10). None was tested as a change for this post.
- **The frozen build.** It differs from earlier builds by several changes at once, among them shared instructions to check a confirmed mechanism's sibling sites and consequences, more passes at high reasoning and a per-file coverage check. Its lower miss count does not isolate any one of them.
### 13. The frozen build's results and costs (GPT-6.1 Sol)
- **Results.** On the 185 pull requests, 461 of 520 defects (88.7%), counting a partial find as half.
- **Costs.** $2.06 a pull request on average, 4 to 31 times the six other tools. The ratios come from unrounded costs. The rounded $2.06 and $0.07 would give 29.
- **What it posts.** 6.9 comments a pull request.
### 14. Tuning and conflicts of interest
- **Who did this.** I build QuickReview and wrote this post. The assistant that designed QuickReview's benchmark-driven changes also orchestrated the key, wrote its admission rule and wrote the judge prompt.
- **Tuning.** The September build's benchmark-driven changes were designed from its misses on 40 of the 185 pull requests. The frozen build's last tuning round was mined from misses on 145 of the 185.
- **Model families.** The judge, GPT-6 Sol, is the model the September run reviewed with, and the frozen build's GPT-6.1 Sol is from the same family. The key was built by Claude agents, the same family as Claude Code's model and the second judge.
### 15. What is verified, and what is my recount
- **The data.** Mateusz Jakubczak, *Real defects in code review: QuickReview benchmark data*, Quickchat AI, 2026, on Hugging Face at [quickchatai/quickreview-bench](https://huggingface.co/datasets/quickchatai/quickreview-bench), DOI [10.57967/hf/11598](https://doi.org/10.57967/hf/11598), licence Apache-2.0. It is public and needs no login. It holds the key, every review, every judge verdict and the run directories. The code that built and checked it is the repository at the tag [`bench-data-v1`](https://github.com/quickchatai/quickreview/tree/bench-data-v1).
- **Verified.** The repository's claims check (`bench/aacr/claims.json`, run by `bench/aacr/verify_claims.py`) recomputes 6,337 numbers printed in the papers, the README, the method documents and the dataset card from that data, and all of them pass. That covers the 80 outright misses, the validator figures and dispositions, the run totals and every reviewer's found count.
- **My recount.** The rest are my own recounts for this post, computed from the same dataset. They first reproduced the verified counts exactly, then split them further. That covers every count of comments inside a defect's lines, the chance placements, the second and third judges' verdicts on the misses, the independent-source subset, what the comments on top were, the miss rates under a comment, the defect-by-defect comparison of low and default, and the re-run. Most of them need only four of the dataset's tables: `defects`, `reviews`, `defect_verdicts` and `comment_verdicts`. The third judge's verdicts are in the dataset's `second_judges/` folder. The chance placements also need each pull request's diff, which the commits named in the `prs` table give.
- **What is not checked for you.** The claims check does not cover my recounts, and the scripts behind them are not in the public repository.
---
Not answered on this page? Ask Quickchat AI directly: https://app.quickchat.ai/v1/api_core/public-question/quickchat/ask/?q=your+question&placement=www-markdown-footer-ask-v1&format=html