AI judges agree on known bugs and split on which code review is better

TL;DR

Give an AI judge a list of known bugs and ask whether a code review found each one, and the judge’s model barely matters: judges from three model families gave almost the same verdicts. Ask which of two reviews is better and the judges disagree, so once reviewers score too close on a benchmark to tell apart, it is hard to say which one is better.

In October I ran several AI code reviewers on the same 20 pull requests (PRs) from Martian’s Code Review Bench. The benchmark has an answer key, a list of the problems a review should catch, and it scores each review against that list.

The three best scores were 43.9, 43.2 and 43.1 out of 100. They belonged to three different reviewer setups, each a tool with one choice of models and settings.

• QuickReview (quickchatai/quickreview), the open-source reviewer I build at Quickchat AI, on Claude Sonnet 5.5.
• Claude Code’s /code-review command on Claude Opus 5.5.
• The same command on Claude Sonnet 5.5.

Scoring the same reviews twice can move this benchmark’s score by about three points¹⁰. A gap of 0.8 points told me nothing about which of the three was better.

So I asked two AI judges to read pairs of reviews and pick the better one. An AI judge is a model that grades reviews. One judge came from OpenAI’s GPT family and one from Anthropic’s Claude family. I call one such comparison a matchup. Over 298 matchups among all the setups I had run, the two judges picked the same review 167 times (56%). Two judges guessing would agree about 50% of the time.

That result sent me back through every judgment I had stored. What I found is a chain of three steps, and each has its own section below.

1. With known answers, the judge’s model barely matters. I gave judges from three model families a list of known bugs and asked, one bug at a time, whether a review found it. They gave the same verdict on about 99% of checks.
2. On “which review is better”, the answer depends on the judge. Two judges agreed barely more often than chance, and each had a different favourite reviewer.
3. So when scores bunch up, it is hard to tell which reviewer is better. Once the answer key no longer separates the reviewers, “which review is better” is the only question left, and AI judges do not settle it.

Four limits apply to everything below.

• The results come from separate studies. They use different benchmarks, different judge designs and different GPT versions¹. The GPT judge is GPT-6 Sol for the bug checks and GPT-6.1 Sol for the review picks.
• Agents built the answer key for the bug checks. Claude agents verified each bug in the code. QuickReview was the key’s largest single source. No human has checked the key or any judge yet.
• The 298 matchups are a small, chosen set. They cover 20 PRs, chosen where QuickReview had done worst. They compare eight reviewer setups, and five are QuickReview’s.
• I build one of the reviewers being judged, and it is the GPT judge’s favourite.

With known bugs to check against, judges from three model families gave the same verdicts

The bug checks run on AACR-Bench, a public benchmark of PRs. For the 185 PRs that can still be run, I had Claude agents build a list of 520 bugs. Each bug is verified in the code and has a note on what counts as finding it². That list is the answer key for this benchmark. The key, the reviews and every judge’s verdict are public in the quickreview-bench dataset on Hugging Face (DOI 10.57967/hf/11598), and every count in this post can be recomputed from it.

The judge takes one review and marks every bug on the PR’s answer key as found, partly found or missed. Each mark is one check.

My first judge was GPT-6 Sol. I then ran the same prompt on Claude Opus 5.5 over the same 1,295 reviews by seven reviewers. The two judges gave the same verdict on 3,593 of 3,640 checks (98.7%). Of the 47 disagreements, 45 were over partial credit, and only 2 set a “found” against a “missed”³. After rounding, every reviewer’s share of bugs found came out the same under both judges.

The third judge was DeepSeek V4.1 Flash, an open model. It graded a later set of 1,168 reviews by seven reviewers, which the GPT-6 Sol judge had also graded. It worked with a handicap. The GPT judge read the repository while it judged. The DeepSeek judge had the same prompt with no tools and no repository, so it judged from the prompt alone. It still gave the same verdict as the GPT judge on 3,601 of 3,640 checks (98.9%). No reviewer’s share of bugs found moved by more than 0.4 points, and the order of the seven stayed the same¹⁷.

A re-run gives the scale. I ran the GPT judge a second time on 210 of the reviews, and it changed 2 of its own 637 verdicts. On the same 637, switching to the Claude judge changed 5.

For this kind of question the three judges were interchangeable. The work that decides the result is the answer key, and mine still needs a human check.

Asked which review is better, two judges agreed on 167 of 298 matchups

The matchups come from my October post on Claude Code’s review. The judge read two reviews of the same PR side by side, with the diff, the changed files and the benchmark’s answer key. It picked the one a maintainer would rather receive⁴.

GPT-6.1 Sol and Claude Opus 5.5 each judged the same 298 matchups with the same prompt. The matchups were among eight reviewer setups. Three were setups of Claude Code’s local /code-review command, and five were QuickReview on different models¹².

The judges named the same winner in 167. Two judges guessing would agree on about 149⁵. The matchups rest on only 20 PRs, and resampling those PRs puts the agreement anywhere from 48% to 64%, a range that includes chance. Neither judge favoured the review shown first⁸.

Each judge’s best record of the eight went to a different setup⁶.

With the GPT judge alone, I would have reported my own tool as the winner⁹. With the Claude judge alone, I would have reported it second from last.

In about half of their disagreements, the judges had read the two reviews the same way and weighed them differently. In 122 matchups both agreed that one review found more and the other wasted less of the author’s time. The Claude judge then picked the review that found more in 111 of them, and the GPT judge in 59⁷. How much a missed bug weighs against a wasted minute is a question of taste. My prompt left it open, and each model answered it in its own way. Each judge’s favourite also ran on its own model family, and I cannot rule a family preference out.

Once the scores bunch up, “which review is better” is the only question left

An answer key separates reviewers only while their scores differ by more than the benchmark’s noise. People call a benchmark saturated when the best scores are close to the maximum. I use the word a little more widely here, for any set of reviewers whose scores are too close for the benchmark to separate. I have three cases, and each is small.

The three reviewers from the opening. The benchmark’s own score, called F2¹⁵, put them within 0.8 points. The two judges compared their reviews directly in 33 matchups and named the same winner in 18. QuickReview on Sonnet 5.5 won 15 of its 25 matchups under the GPT judge and 6 under the Claude judge¹⁸.

The answer key could not rank these three. The GPT judge put QuickReview first of the three, and the Claude judge put it last.

I still do not know which of the three is best. These scores bunch up far below the maximum. Martian’s answer key is short, about 3.7 entries per PR, and it counts a true finding it lacks the same as a wrong one.

The two top tools on Martian’s full 50 PRs. Of the seven tools I scored there, qodo-extended-v2 leads with 59.3 and cubic-v2 follows with 58.9 under the GPT judge. Under the DeepSeek judge they score 60.7 and 61.1, and they swap places¹⁸. A change of judge that moves a score by a point or two does no harm when reviewers are ten points apart. It decides the order when they are half a point apart.

The serious bugs on AACR-Bench. This is the one case of saturation in the usual sense. The answer key has 90 high-severity bugs. The QuickReview build I froze on 4 October finds 87.5 of them, Codex CLI finds 81 and the pi-review panel finds 80¹¹. A reviewer better than mine could gain at most 2.5 bugs on that list. Over all 520 bugs the same three are far apart, at 89%, 55% and 67%, so the benchmark as a whole still separates them.

This is where steps 1 and 2 meet. While an answer key separates the reviewers, the three judges I tried are interchangeable. When it stops separating them, the question becomes “which review is better”, and a GPT judge and a Claude judge gave me different answers to it.

What I do about it

Toufique Ahmed and colleagues propose agreement between models “as a predictor of whether a given task is suitable for LLMs at all”. By that test, checking a review against known bugs suits a model. Picking the better review, asked the way I asked it, does not.

• Stay on known answers for as long as they separate the reviewers. Give the judge a list of bugs verified in the code and ask about each bug in turn.
• When the scores bunch up, build a harder benchmark. More PRs and harder bugs give the answer key room to separate reviewers again.
• Put the care into the answer key. Judges that agree may share a mistake¹³, and only a person can check the key. Mine is still waiting. A blinded audit of 20 bugs and 5 rejected candidates is ready for an engineer.
• If you must ask which review is better, name the judge. Use judges from more than one model family, publish each judge’s table and say where they split¹⁶. Write your own weighting of missed bugs against wasted comments into the prompt.
• Or put a person in the loop. A maintainer reading both reviews of a few of their own PRs answers the question for their team.

What this does not show

• Agreement is not correctness. Judges that agree have read the reviews the same way. Whether they read them right is a question for a human check, which has not happened yet.
• The studies differ in more than the question. I read the kind of question as the likeliest cause of the gap, and I cannot prove it¹.
• Step 2 rests on two judges and one prompt. The prompt left the weighting open. A prompt that fixes it might bring the judges closer, and I have not tested that. The judges were not re-run.
• Step 3 rests on small numbers. The three close reviewers share 33 matchups, and the high-severity list has 90 bugs.
• One project made everything. The same project built the reviewer, the answer key and the judge prompts¹⁴.

Frequently asked questions

Can an LLM judge be trusted to grade AI code reviews?

It depends on the question. Given a list of known bugs and asked whether a review found each one, judges from three model families gave nearly the same verdicts. A GPT-6 Sol judge and a Claude Opus 5.5 judge agreed on 3,593 of 3,640 checks (98.7%). A DeepSeek V4.1 Flash judge agreed with the GPT judge on 3,601 of 3,640 checks (98.9%) on a later set of reviews. Asked which of two reviews was better, a GPT-6.1 Sol judge and a Claude Opus 5.5 judge agreed on 167 of 298 matchups (56%), where chance gives about 50%. These results come from separate studies. Agreement does not show that any judge is right, and no human has checked them yet.

Why do two AI judges disagree about which code review is better?

In about half of their disagreements, the two judges had read the reviews the same way and weighed them differently. In 122 matchups both judges agreed that one review found more and the other wasted less of the author’s time. The Claude Opus 5.5 judge then picked the review that found more in 111 of them, and the GPT-6.1 Sol judge in 59. My prompt did not say which of the two matters more, so each model answered that in its own way. Each judge’s favourite reviewer also ran on its own model family, and my data cannot rule a family preference out.

What happens when a code review benchmark is saturated?

The score stops telling the reviewers apart, and the only question left is which review is better. AI judges do not settle that question. On 20 pull requests of Martian’s Code Review Bench, three reviewer setups scored 43.9, 43.2 and 43.1 out of 100. Two AI judges then compared their reviews directly and named the same winner in 18 of 33 matchups. QuickReview on Sonnet 5.5 won 15 of its 25 matchups under the GPT judge and 6 under the Claude judge. On the 90 high-severity bugs of my AACR-Bench answer key, QuickReview finds 87.5, so a better reviewer could gain at most 2.5 bugs there.

How should I grade AI code reviewers with an LLM judge?

Use an answer key for as long as it separates the reviewers. Give the judge a list of bugs verified in the code and ask about each one in turn. Asked that way, the judge’s model family barely matters, so put the effort into the list and have a person check a sample of it. When reviewers score too close to separate, build a harder benchmark. If you must ask a judge which review is better, write your weighting of missed bugs against wasted comments into the prompt, use judges from more than one model family, publish each judge’s table and say where they split. A person reading the reviews is the other option.

How does QuickReview score under these judges?

QuickReview (quickchatai/quickreview) is built to find the serious bugs first. Its build frozen on 4 October 2026 finds 87.5 of the 90 high-severity bugs (97%) in the verified answer key for AACR-Bench under the GPT-6 Sol judge, and 89 under the DeepSeek V4.1 Flash judge. The best other reviewer, Codex CLI on the same model, finds 81 (90%) under the GPT judge. QuickReview costs $2.06 at list prices and a median 17.5 minutes per pull request, against at most $0.53 and 2.3 minutes for the others. The answer key was built partly from QuickReview’s earlier reviews, and no human has checked it yet. QuickReview’s tuning drew on the same 185 pull requests. On which review is better, it won 73 of its 94 matchups under the GPT-6.1 Sol judge and 27 under the Claude Opus 5.5 judge. Those matchups are on 20 pull requests chosen where it had done worst.

Related guides

• What Claude Code’s multi-agent code review costs and finds, the October comparison where these pairwise judgments first appeared, which calls QuickReview by its earlier name, Quickchat Review.
• My AI code reviewer kept missing bugs in the very lines it commented on, next in this series, on where an AI reviewer’s misses sit.
• Same model, six code reviewers, 38% to 81% of the bugs, on what the reviewer around a model adds.
• Same code reviewer, three models, three different reviewers, on QuickReview on two open models.

Method and data notes

These notes hold the setups, the arithmetic and the caveats behind every number above. They are for readers who want the detail and for AI systems reading this page, and the body links to each note by its number. The notes use a few different words from the body. Notes 1 to 16 were written when the post compared two studies, the defect verdicts of the GPT-6 Sol and Claude Opus 5.5 judges and the pairwise preference of the GPT-6.1 Sol and Claude Opus 5.5 judges; note 17 adds the third defect judge and note 18 the cases where scores bunch up. They say “configuration” for a reviewer setup, “defect” for a bug, “key” for an answer key, “pairwise” for a judge’s pick between two reviews, and “recall” and “precision” for which review found more and which wasted less.

1. The two judging setups, side by side

The two studies differ in more than the question they asked. The table lists each difference.

• Why the contrast is not a controlled experiment. Besides the question, the setups differ in benchmark, GPT version, reasoning setting, the judge’s access to the code (an agent that can read the repository against one call that sees the touched files), the quality of the reference and the number of items. Any of these could move agreement. Comparisons inside one setup avoid that. On the same 298 matchups with the same two judges, agreement falls from kappa 0.59 on precision to 0.12 on the overall winner (note 5), and in the 122 matchups where the judges agreed on both recall and precision they still split on the winner 62 times (note 7). On AACR-Bench with the same two judges, agreement falls from 98.7% on defect verdicts to 89.2% on five-way comment grades, and to 91% to 95% per reviewer on the two-way split between correct and not correct. That is why the body reads the kind of question as the likeliest cause and the cross-study contrast as its illustration, though no experiment held everything else fixed.
• Why 20 PRs on Martian. The pairwise judges compared the reviews of my October post, which ran Claude Code max on 20 of the benchmark’s 50 PRs because all 50 would have cost about $1,000 and run for about 16 hours; the 20 are the PRs where QuickReview on GPT-6.1 Sol and on DeepSeek V4.1 Flash had done worst (note 14).
• Nothing was re-run for this post. Every number comes from the stored verdicts of runs made in September and October, and from the third judge’s run of 2026-11-09 for the data release (note 17); the new counts in notes 3, 5 to 9, 17 and 18 are arithmetic on those verdicts and on the stored reviews, all of them in the public dataset (note 16).

2. The AACR-Bench key and how it was verified

• AACR-Bench. Version 1.0 holds 200 pull requests from 50 public repositories. Its harness scores 196, and 11 of those could not be run because GitHub no longer serves their head commit, which leaves 185 PRs in ten languages. Its answer key is a list of reference comments; by the dataset’s own account 74% were generated by LLMs and kept by annotators, and they cover code defects, maintainability and readability, performance and security.
• What the references are. Agent-made labels of all 253 references on 30 of the PRs found 29 real defects (11%), 20 real maintainability issues (8%), 77 trivial (30%), 99 dubious or wrong (39%) and 28 duplicates (11%). During key construction on 145 other PRs, every one of their 1,107 references was checked against the code as a candidate defect, and 142 (12.8%) turned out to be part of a real defect. The rest were rejected as wrong, trivial, style, pre-existing or unreachable, each with a recorded reason. Over all 185 PRs, only 162 of the 520 verified defects have an AACR reference among their sources. A real defect the PR introduced is a stricter test than a review comment has to pass, and these labels and checks are agents’ work with no human check.
• How the key was built. Candidates came from an expert hunt by Claude agents, the references, and reviewers’ comments (three reviewers first, four more in an extension). Verifier agents checked every candidate against the code with the reviewer comments anonymised and shuffled, and critic agents spot-checked the result and fixed the key in place. The result is 520 defects (90 high, 145 medium, 285 low severity) and 3,093 rejected candidates kept with their reasons. Claude Code subagents on Opus 5.5 built and critiqued the key, and the same assistant that designed QuickReview’s changes for this benchmark orchestrated the key and wrote the judge prompt.
• QuickReview as a source. QuickReview’s September build at default settings, on GPT-6 Sol, was the largest single contributor to the first version of the key for 145 PRs, raising 334 of its 379 defects (88%) and 29 that no other source raised. The paper measures what that could do. On the 369 defects (82 of them high severity) that the expert hunt or a reference raised, which would be in the key had no reviewer comment been a candidate, that build finds 86% under both judges and 80 of the 82 high-severity ones, against 70% for Claude Code, 61% and 60% for pi-review and 59% for open-code-review; in the earlier study a leave-one-reviewer-out key moved no reviewer’s recall by more than 0.8 points.
• The headline build on the same subset. The build frozen on 4 October was never a source for the key, since the comments of the final comparison’s GPT-6.1 Sol runs were never candidates. Scored by the primary judge from the stored verdicts, it finds 332 of the 369 (90%) and 81 of the 82 high-severity ones, against 260 (70%) and 69.5 for Claude Code and 222 (60%) and 76 for Codex CLI; the paper reports this check for the compared build of 1 October, and this row is computed here with the same rule, which reproduces the paper’s figure for every other reviewer. Three cautions apply to this subset. Every reviewer scores higher on it, and it leans towards defects easier to see from the diff. Each defect’s wording, severity and “counts if” note were written with every candidate reviewer’s comments in view. The frozen build’s last tuning round drew on misses from 145 of these PRs.
• No human check yet. No human has checked any part of the key, and no sample of either judge’s verdicts has been graded by a person. A blinded audit of 20 defects on Python PRs and 5 exclusions, with full context per item, is ready for one engineer.

3. The defect judges’ agreement in full

Under both judges, each reviewer found nearly the same number of the 520 defects.

The paper’s earlier study, 28 to 30 September, with every third-party tool as shipped or as AACR-Bench’s harness runs it, none of them given the PR’s title and description, pi-review and OpenCode below their best configurations, and Claude Code at its session default, whose 62% the paper calls not a best-shot figure (note 12). A partial find counts as half.

• Judges and inputs. The primary judge is GPT-6 Sol at high reasoning. The same prompt, byte for byte, with the same inputs and parser, was run a second time on Claude Opus 5.5 over all 1,295 reviews of the earlier study. The final comparison’s reviews, which carry QuickReview’s headline numbers, were graded by the primary judge. The Opus judge did not grade them. For the data release the DeepSeek judge graded them again (note 17).
• Per defect. The same verdict on 3,593 of 3,640 reviewer-defect pairs (98.7%) and on 628 of 630 high-severity pairs. Cohen’s kappa is 0.974 on found, partial and missed, and 0.986 when found and partial are merged (99.3% raw). Of the 47 disagreements, 45 involve a partial on one side and 2 are found against missed. Problems found moves by at most 3 defects per reviewer, and recall is identical after rounding for all seven: 84%, 66%, 62%, 56%, 56%, 48% and 48% in the table’s order.
• Clear calls and partial calls. Recounted from the stored verdict files, with the GPT-6 Sol judge’s verdict first and the Opus 5.5 judge’s second: found and found 2,146, found and partial 15, found and missed 2; partial and found 8, partial and partial 33, partial and missed 12; missed and found 0, missed and partial 10, missed and missed 1,414. On the 3,562 pairs where neither judge said partial they agree on 3,560; on the 78 where either did, on 33. A band defined by either judge’s partial holds every partial disagreement by construction, so the 33 of 78 describes the band and is no estimate of how often the judges agree on a hard call. High-severity pairs drew a partial from either judge 8 times, and the judges agreed on 6. A partial counts half, so this band moves no reviewer’s problems found by more than 3 defects, and it is the band a human audit should sample first.
• The ranking. pi-review and open-code-review, 1 and 1.5 defects apart under the two judges, swap places; Kendall’s tau over the seven is 0.90, one discordant pair of 21.
• Self-consistency. The primary judge was re-run once on 30 random PRs with all seven reviewers (210 reviews, 637 reviewer-defect pairs, 742 comments). It gave the same defect verdict on 99.7% of pairs (kappa 0.994), 635 of the 637, and both of its disagreements involve a partial. On the same sample the two judges agree on 99.2% (kappa 0.984), 632 of the 637.
• Per comment. The same five-way grade (correct and valuable, correct but minor, partly correct, incorrect, unverifiable) on 3,972 of 4,455 comments (89.2%), and the same split between correct and not correct on 91% to 95% of each reviewer’s comments. On the re-run sample the primary judge matched itself on 92.2% of grades (kappa 0.87) against 88.4% between the judges, so 7.8 of the 11.6 points of disagreement between the judges, about two-thirds, are at the level of one judge’s disagreement with itself.
• The same-family effect. Comments correct differ by up to 5 points between judges. The largest gaps are Claude Code (81% under GPT, 86% under Opus) and open-code-review, an OpenAI-based tool (73% and 77%), so the Claude judge was also kinder to a tool outside its family; Claude Code’s recall is 62% under both. The paper reads this as the one visible same-family effect, and cites the information-retrieval finding that assessor disagreement changes absolute scores more than system rankings (Voorhees, 2000).
• The earlier study’s tools. The seven reviewers ran on GPT-6 Sol except Claude Code 2.1.283 (/code-review on Opus 5.5 at the session default effort, medium), with the four third-party OpenAI-based tools at medium reasoning, as shipped or as the AACR harness runs them. The paper’s fairness audit of those runs found three ways they disfavoured the third-party tools. pi-review ran below its own default reasoning level and without its panel, OpenCode ran through a compatibility bridge instead of its own provider, and no tool but QuickReview was given the PR title and body. The paper’s final comparison enters each third-party tool except Claude Code in the best configuration found. The earlier study is used here because every two-judge check was run on it.
• Cost. Judging the 1,295 reviews once cost $41.85 at list price with GPT-6 Sol and $54.16 with Opus 5.5, or $0.035 and $0.045 for each of the 1,199 reviews that needed a call; the 96 with no defect and no comment needed none.

4. The pairwise judge

• One judgment is one model call, with no tools and no agent loop, and structured output accepted only as one plain final message.
• The prompt, in order. The PR’s title and description, its diff, every file it touches as it is at the head commit, line-numbered, Martian’s answer-key comments, then Review A and Review B. Lockfiles, checksum files, minified bundles and source maps are replaced by one line with their changed-line counts. The judge sees the touched files only, not the callers, dependencies or tests a reviewer could open.
• Blinding. Both reviews are rewritten into one neutral shape (path, lines, text and any stated severity), with QuickReview’s pass labels and signature stripped and Claude Code’s dump of its own findings removed. A and B order is random, drawn once per pair, and no pair was judged in both orders. A stated severity, the number of findings and the writing style remain, so a judge may still infer which tool wrote a review.
• The verdict. Four categories, each with reasoning and then A, B or tie. Recall asks which review catches more of the real defects that matter, the key’s comments plus real defects the key lacks, weighted by severity. Precision asks which spends less of the author’s attention on findings that are wrong, unreachable or not worth raising. Finding quality asks which states its real findings better, with a correct severity, a precise location, a concrete failure scenario, the root cause and a fix the author can act on. Writing asks which a developer can read and act on faster. The overall question asks which review a maintainer of this repository would rather have received, with the instruction “Catching the serious defects and not wasting the author’s time matter most; finding quality next; writing can decide a close call.” The judge is told that more findings, longer text and confident tone earn nothing on their own. A tie is allowed in any category and overall “when the difference is too small to matter, or when the reviews are better in offsetting ways you cannot rank”; neither judge called an overall tie in 298 matchups.
• Judges and sampling. GPT-6.1 Sol and Claude Opus 5.5, both at xhigh, with an identical prompt. Round 0 judged every pair of the eight configurations on the same 5 PRs (140 judgments per judge); an adaptive round gave 3 fresh PRs to each of 6 neighbouring pairs whose gap was unresolved (18); a final round judged every pair among the top five on every remaining PR (140). That makes 298 per judge over all 20 PRs, under a cap of 300 per judge. 28 GPT judgments made before a change in how lockfiles were packed were set aside and the pairs judged again, so both judges’ verdicts rest on the same prompts.
• Ratings. A Bradley-Terry fit per judge, shown on the Elo scale, with 95% intervals from a bootstrap over PRs. The pooled table counts each matchup once per judge (596 games), at equal weight, a rule I chose. The full method and every interval are in notes 10 to 12 of the October post.

5. How kappa, tau and chance agreement were computed

• Cohen’s kappa (Cohen, 1960) is (p_o − p_e) / (1 − p_e), where p_o is the observed agreement and p_e the agreement two independent raters with the same answer rates would reach. The bands commonly used to read it (Landis and Koch, 1977) call 0.81 to 1.00 almost perfect, 0.41 to 0.60 moderate and 0.00 to 0.20 slight. Raw agreement on unbalanced answers can be high while kappa is low (Feinstein and Cicchetti, 1990), which is why the paper reports kappa beside the 98.7%.
• The pairwise overall verdict. The GPT judge picked Review A in 152 of the 298 matchups and the Opus judge in 169, so p_e = (152 × 169 + 146 × 129) / 298² = 50.1%. Observed agreement is 167 of 298 (56.0%), so kappa = 0.12. Chance alone would give about 149 agreements. The 298 matchups share 20 PRs and eight configurations, so they are not independent draws. Resampling the 20 PRs 10,000 times puts agreement between 47.7% and 64.4% and kappa between −0.05 and 0.29 (95%), with kappa at or below zero in 8% of resamples. Agreement per PR ranges from 1 of 6 matchups to 10 of 10, and 7 of the 20 PRs are at or below 50%.
• The pairwise categories, with the tie as a third answer, so a tie from both judges counts as the same verdict: same verdict on recall in 210 of 298 (kappa 0.53), precision 226 (kappa 0.59), finding quality 148 (kappa 0.18) and writing 220 (kappa 0.50). Opposite winners in 21, 20, 76 and 61; in the rest of each category one judge called a tie. Counting only matchups where both judges named a winner, they picked the same review for recall in 189 of 210, precision in 221 of 241, finding quality in 142 of 218 and writing in 220 of 281; the overall question, which neither judge ever answered with a tie, in 167 of 298.
• The defect verdicts. The paper gives p_o = 3,593 / 3,640 = 98.71% and kappa = 0.974, which implies p_e = (0.98709 − 0.974) / (1 − 0.974), about 50%; with kappa anywhere from 0.9735 to 0.9745 it is between 49% and 51%.
• Kendall’s tau is (concordant − discordant) / pairs, counting the pairs of configurations that two rankings put in the same and in the opposite order. Overall pairwise ratings, from each judge’s published column: 14 concordant and 14 discordant of 28, tau 0.00. Per category, from each judge’s per-category ratings: recall 23 and 5 (tau 0.64), precision 25 and 3 (0.79), finding quality 20 and 8 (0.43), writing 22 and 6 (0.57). AACR-Bench: 20 and 1 of 21 (0.90). The seven AACR-Bench reviewers spread from 48% to 84% recall, while under the Opus judge five Martian configurations sit within 64 Elo points, so the two taus also compare a spread field with a bunched one.
• Reproducing the published rates. Recomputed from the stored verdicts, the same-winner counts match the published agreement rates exactly (167, 210, 226, 148 and 220 of 298).

6. Each judge’s table

Each judge’s ratings are fitted on its own. Both judges answered the same 298 matchups, so a configuration’s wins under each judge are over the same games. Claims per PR is Martian’s extraction of each review into claims, worked out from the October post’s score table as unmatched claims divided by one minus precision, averaged over the 20 PRs.

• Point estimates. Ranks are point estimates. Under the Opus judge, places 4 to 8 lie within 64 points (1340 to 1404), and seventh place is 2 points below sixth, so “seventh” means the bottom group; under the GPT judge, QuickReview on GPT-6.1 Sol leads the second place by +140, with a 95% interval of +13 to +302. The three configurations that were not in the final round rest mostly on round 0 and have wide intervals.
• Per-category places. Both judges put max first on recall and QuickReview on Sonnet 5.5 first on precision and writing and last on recall. They differ most on QuickReview on GPT-6.1 Sol, 3rd and 7th on recall, 2nd and 4th on precision, and 1st and 6th on finding quality.
• Swing and volume. The change in win share from the GPT judge to the Opus judge has a Spearman rank correlation of 0.81 with claims per PR over the eight configurations, and the same with unmatched claims per PR. QuickReview on Sonnet 5.5 made the fewest unmatched claims, 6.2 per PR. The largest exception is QuickReview on GPT-6.1 Sol, a review of middling length that lost the most, which is where a family preference of the GPT judge would show.
• Selection. QuickReview’s GPT-6.1 Sol and DeepSeek V4.1 Flash reviews here are the reviews whose missed answer-key issues chose the 20 PRs, so on these PRs they are those builds’ weakest reviews by the key. A judge that follows recall is expected to rate them low, which accounts for part of the GPT-6.1 Sol build’s fall under the Opus judge, beside any family preference of the GPT judge.

7. The trade-off matchups and the judges’ readings

• Definition. A trade-off matchup is one where both judges named the same recall winner and the same precision winner, neither was a tie, and the two winners were different reviews. “Found more” is the recall winner and “wasted less” the precision winner, in the prompt’s own terms (note 4).
• All 122. Both judges took the review that found more in 54, both took the one that wasted less in 6, the GPT judge took the one that wasted less and the Opus judge the one that found more in 57, and the reverse in 5. Same winner 60, opposite 62.
• By the families behind the two reviews. Both reviews from configurations on Claude models: 74 matchups, with 35, 4, 34 and 1 in the same four cells, so the Opus judge took the review that found more in 69 and the GPT judge in 36. Of the 74, 61 include QuickReview on Sonnet 5.5 or on Opus 5.5 (the Opus judge 56, the GPT judge 26) and 13 are between two Claude Code configurations (13 and 10), too few to size the gap on their own. One review from QuickReview on GPT-6.1 Sol: 34 matchups, 32 of them against a configuration on a Claude model and 2 against an open model (Opus 31, GPT 19). The other 14 have an open model on one side, 12 against a configuration on a Claude model and 2 of DeepSeek V4.1 Flash against GLM-5.3 Flash (Opus 11, GPT 4). QuickReview on Sonnet 5.5 and on Opus 5.5 ran six code-quality passes on GPT-6 Luna (31% of the pass runs in the Sonnet 5.5 configuration, under 2% of either configuration’s cost); every other pass, the validator and the session that writes the review ran on the Claude model.
• The three readings. In 66 matchups each judge’s recall and precision verdicts favoured one review, the other verdict naming the same review or a tie, and it was the same review for both judges: same overall winner 63 (in the 23 where both judges named that review the winner of both categories, 23). In the 122 trade-offs, 60. In the other 110 the two judges’ readings differed, and they named the same winner in 44. The 131 opposite overall winners are therefore 3, 62 and 66. Counted another way, 64 of the 131 had identical recall and precision verdicts from both judges, ties included, 62 of them trade-offs.
• Each judge on its own. Over every matchup where a judge’s own recall and precision winners were different reviews, the Opus judge’s overall pick followed recall in 143 of 168 (85%) and the GPT judge’s in 79 of 157 (50%).
• Recall or length. Finding counts are each review’s inline comments in its stored Martian file. In 121 of the 122 trade-off matchups the review that found more also posted more findings (the 122nd posted the same number), so inside them a judge that weighs recall and a judge that likes longer reviews make the same pick. In the 22 matchups where both judges named the review with fewer findings as the recall winner, the Opus judge picked that review all 22 times and the GPT judge 21; in 13 of the 22 both judges also named it the precision winner.
• One matchup as an example. Two Claude Code reviews of a Discourse PR that shrinks large image uploads. Both judges said the longer review found more: it caught all four issues in the answer key and the shorter one caught three. Both also said the shorter review wasted less of the author’s time; the Opus judge counted 10 findings in it and 15 in the longer one. The GPT judge then picked the shorter review, because the longer one’s extra findings “do not outweigh A’s much larger attention cost”, and the Opus judge picked the longer one “for its complete coverage of the golden issues and stronger root-cause analysis, despite its verbosity” (A is the longer review).
• What this shows. The pattern fits a difference in how the two judges weigh recall against precision, and the subset with two Claude-made reviews shows it without a family difference between the reviews. It does not rule out a family preference acting elsewhere, and the analyses were defined after the judgments were made.

8. Close calls, noise and position

• By round. Same overall winner in 70 of 140 matchups in round 0 (every pair, the same 5 PRs; 50%), 13 of 18 in the adaptive round (72%) and 84 of 140 in the final round among the top five (60%). Round 0’s five PRs gave 19, 16, 16, 11 and 8 agreements of 28. The adaptive round was aimed at neighbouring pairs whose rating gap was still unresolved, and the final round at every pair among the top five.
• By rating gap. Same overall winner in 66 of 127 matchups between configurations under 100 pooled Elo points apart, 72 of 130 at 100 to 199 and 29 of 41 at 200 or more. The pooled ratings are fitted on these same verdicts, which pulls the configurations the judges disagree on closer together, so the figure for large gaps flatters agreement.
• Two small repeats. Before the full run, 12 round-0 Opus judgments on two PRs were made live, and the full run made them again through the Batch API with the identical prompt; they named the same overall winner in 11 of 12, against 5 of 12 between the two judges on those matchups. The 28 GPT judgments set aside after the packing change were all on one PR; judged again with the new context, the GPT judge picked the same overall winner in all 28, and it agreed with the Opus judge on 8 of them. Both are small and the second rests on a changed prompt, so they are weak checks, and no pairwise judge was re-run by design.
• Position. Under each judge’s own ratings, Review A would be expected to win 150.3 matchups under GPT and 163.5 under Opus; it won 152 and 169, so neither judge shows a measurable preference for the first review. Position, verbosity and self-preference are known biases of LLM judges (Zheng et al., 2023; Panickssery et al., 2024).

9. The early read with one judge

• Round 0 under the GPT judge alone. The design called for both judges, rated separately with an identical prompt, and the GPT judge ran round 0 alone first, 140 judgments over 5 PRs. Its ratings then were QuickReview on GPT-6.1 Sol 1890, on Sonnet 5.5 1756, on Opus 5.5 1559, Claude Code high on Opus 5.5 1434, high on Sonnet 5.5 1394, QuickReview on DeepSeek V4.1 Flash 1373, Claude Code max 1353 and QuickReview on GLM-5.3 Flash 1240. The same Bradley-Terry fit on the stored round-0 judgments gives the same eight figures. These provisional figures from 5 PRs were superseded by the full run, and are quoted only to show what a single judge would have reported.

10. The answer key against the judges on Martian

• The benchmark’s score. Martian’s pipeline splits each review into claims and matches them by meaning against the golden comments, about 3.7 per PR (74 on these 20 PRs). On these 20 PRs QuickReview on Sonnet 5.5 scored 43.9 on F2, Claude Code high on Opus 5.5 43.2 and on Sonnet 5.5 43.1, three setups within 0.8 points, and Claude Code max scored last at 30.0. The October post treats about two answer-key issues per configuration as this benchmark’s noise on 20 PRs, and the paper measured about three points of judge noise between two scoring passes of the same reviews on Martian’s benchmark. A claim missing from a key this short counts against precision whether it is true or wrong.
• Key and judges. The eight configurations’ orders by F2 and by the pooled judges run against each other (Kendall’s tau −0.36); against each judge alone tau is −0.07 (GPT) and −0.21 (Opus). The review the key scores last is the one the pooled judges rate first, as the October post’s slope chart shows.
• The golden comments. The benchmark paper audited all 173 golden comments of the 50-PR set with the method used for the AACR key, one investigator per comment, Claude Opus 5.5 agents told to make the strongest case that the comment names a real defect the PR introduces, and a second for any verdict other than real: 86 name a real defect the PR introduces, 77 do not and 10 are disputed. No human has checked these verdicts either.

11. QuickReview’s numbers and models

• The frozen build. QuickReview’s build frozen on 2026-10-04, on GPT-6.1 Sol, one review per PR, judged once by the GPT-6 Sol judge. On AACR-Bench it finds 87.5 of 90 high (97%), 134.5 of 145 medium (93%), 239 of 285 low (84%) and 461 of 520 (89%), with partial finds counting half; 92% of its comments are graded correct, at 6.9 comments per PR, 42% of them true but minor; $2.06 per PR at list prices ($380.39 for the 185 PRs) and a median 17.5 minutes. The other tools in the final comparison: the pi-review panel 347.5 (67%), Claude Code on Opus 5.5 324.5 (62%), open-code-review 298.5 (57%), Codex CLI 287 (55%) with 81 of 90 high, OpenCode 263.5 (51%) and PR-Agent 194 (37%), at $0.07 to $0.53 and 0.7 to 2.3 minutes per PR. At those list prices a defect found cost about $0.83 with the frozen build against $0.06 to $0.33 for the other tools.
• The judges behind these numbers. The numbers in this section come from the GPT-6 Sol judge, and the frozen build runs on GPT-6.1 Sol, the same model family. The Opus check was not repeated on these reviews. For the data release a judge from a third family, DeepSeek V4.1 Flash, graded the frozen build and the six third-party reviewers again without the code. It moved no reviewer’s recall by more than 0.4 points and lowered comments correct for six of the seven, the frozen build’s from 92.4% to 85.4% (note 17). On the earlier study’s reviews the Opus judge gave the six GPT-made reviewers 1,861.5 defects in all against the GPT judge’s 1,865 (note 3).
• Tuning disclosure. QuickReview’s September changes were designed from its misses on 40 of the 185 AACR-Bench PRs. The last round before the freeze was mined from misses on the other 145, on 20 Martian PRs and on 59 Qodo issues, and validated on 20 of the 40. On the 20 AACR-Bench PRs it left out, which the September stage had used, the frozen build finds 45.5 of 55 defects (83%) against 39.5 (72%) for the pi-review panel, and QuickReview’s two September builds on GPT-6 Sol found 50.5 and 48.5 of those 55. On the PRs of the other two benchmarks that the last round left out, it ties qodo-extended-v2 on Martian’s real golden comments (43 of 52, 30 PRs) and is level with qodo-exhaustive on Qodo (F1 53.0 against 51.3, 31 PRs).
• In the pairwise run. QuickReview on GPT-6.1 Sol is the frozen build. QuickReview on Opus 5.5 and on Sonnet 5.5 is a development build of 2026-10-19 with six quality passes on GPT-6 Luna ($0.41 of $92.26 and $0.37 of $23.35 over the 20 PRs). The DeepSeek V4.1 Flash and GLM-5.3 Flash rows ran mid-October builds, later than the frozen build and earlier than the development build, in runs of 2026-10-14 and 2026-10-15.

12. Claude Code: which product and version

• Which product. Every Claude Code result here is the local /code-review command bundled with the Claude Code CLI, not Anthropic’s managed Code Review service or Ultrareview.
• On AACR-Bench. It is version 2.1.283 on Opus 5.5 at the session default effort (medium), run as the benchmark’s harness runs it. It was the one tool not given the best-shot search, by my decision. It stayed at its session default, which its documentation says reports only the findings it is most confident in. The harness’s permission set denied its read-only git commands on 46 of the 185 PRs. Its findings were read out of its prose answer by a parser written for this benchmark. The paper calls its 62% not a best-shot figure.
• On Martian. It is version 2.1.303, run headless at high on Opus 5.5 and on Sonnet 5.5, and at max with every subagent forced onto Sonnet 5.5 under an Opus 5.5 main session, which billed $1.74 of that configuration’s $398.03. There max ran under a $30 budget cap its agents could see, five of its 19 finished reviews launched no verifier, and one review that reached the 90-minute limit is absent from the ratings. The setup is in notes 1 to 4 of the October post.

13. What agreement between judges does not show

• Two judges that agree read alike; neither study measures whether they read correctly. Both defect judges read the same notes on what counts, written by the key’s builder, and a shared bias would not show as disagreement. LLM judges have been found to agree with human annotators unevenly across tasks, which is why a human-graded sample is the next measurement.
• Judges from different families are a proposed mitigation for self-preference (Verga et al., 2024), and pooling two of them limits how far either judge’s taste can carry a ranking. It does not remove that taste.

14. Limitations and conflicts of interest

• Different benchmarks, judge designs and GPT versions sit behind the two headline figures, and only the defect judge could read the repository beyond the touched files (note 1).
• No human calibration of either judge, and no human check of the AACR key or of the Martian golden-comment audit.
• The pairwise judges were not re-run on the same matchups by design, beyond the two small repeats in note 8; the kappa, tau, trade-off and reading analyses were defined after the judgments were made; each configuration reviewed each PR once.
• The 20 Martian PRs were chosen, before any Claude Code run, where QuickReview on GPT-6.1 Sol and on DeepSeek V4.1 Flash had done worst, which is expected to favour every other reviewer by an unmeasured amount.
• QuickReview’s last tuning round before the freeze drew on its misses on 20 Martian PRs, and 7 of them are among these 20, so the frozen build on GPT-6.1 Sol had been tuned on part of the set where the GPT judge ranked it first.
• Quickchat AI builds QuickReview, and I built it. The same project built the AACR key, partly from QuickReview’s earlier reviews, and wrote both judge prompts. The GPT judges share a family with QuickReview on GPT models and with six of the seven AACR reviewers; the Claude judge shares one with Claude Code, with QuickReview on Claude models and with the agents that built the key. QuickReview’s shipped default model is OpenAI’s, and Quickchat AI runs on OpenAI and Google Cloud.

15. The benchmarks’ own scores, for comparison

• AACR-Bench’s own metric. Scored against its reference comments in the final comparison, every reviewer recovers between 4.3% and 10.7% of the references, and five reviewers 30 points apart on verified defects (PR-Agent 37% to the pi-review panel 67%) sit within 1.8 points of each other on AACR recall.
• Martian’s own score on these 20 PRs (F2 on Martian’s default Core profile, the 65 of the 74 golden comments on these 20 PRs in its bug, security, concurrency, data, API, performance, test-gap and documentation-defect categories; one GPT-6.1 Sol scoring judge): QuickReview on Sonnet 5.5 43.9, Claude Code high on Opus 5.5 43.2, Claude Code high on Sonnet 5.5 43.1, QuickReview on GLM-5.3 Flash 36.6, QuickReview on Opus 5.5 35.0, QuickReview on DeepSeek V4.1 Flash 34.5, QuickReview on GPT-6.1 Sol 30.8 and Claude Code max 30.0. F2 weights recall four times as heavily as precision; Martian’s documentation describes it as reflecting a cost asymmetry “where missed bugs are worse than false alarms”.

16. Data and code

• The dataset. The answer key, every review and verdict, and both pairwise judges’ verdicts with their reasoning have been public since 2026-11-12, with no login, under Apache-2.0: “Real defects in code review: QuickReview benchmark data”, Hugging Face dataset quickchatai/quickreview-bench, DOI 10.57967/hf/11598. The repository at the tag bench-data-v1 holds the export, the scorers that rebuild the published tables from the dataset without a model call, and both papers. Its CITATION.cff cites the dataset by that DOI.
• Where each kind of verdict is. The defect verdicts of the GPT-6 Sol and Opus 5.5 judges are in the table defect_verdicts, and the pairwise judgments in elo_judgments, with each judge’s ratings in elo_ratings. The third judge’s verdicts are in the dataset’s second_judges folder, beside the tables, with its own README. The pairwise table carries each judge’s reasoning per category; the judges’ overall explanations, which note 7 quotes, are in the run archive’s elo-pairwise volume.
• Counts computed for this post. The counts new in this post are computed from the dataset: the pooled agreement of the third judge (3,601 of 3,640), the kappa values, the trade-off cells, the readings, the rounds, the rating gaps, the position check and the 33 matchups of note 18. They are arithmetic on those tables and on the stored reviews, and the scripts that made them are not in the repository. The repository’s own claims check (bench/aacr/claims.json, run by bench/aacr/verify_claims.py) covers the numbers of the papers, the dataset card and the repository’s pages.
• Papers and code. The defect study’s method, tables and threats to validity are in the benchmark paper and the QuickReview report. The full result tables are in the repository’s BENCHMARKS.md. The pairwise judge’s method and each judge’s ratings are in notes 10 to 12 of the October post, and the code is in the QuickReview repository.

17. The third defect judge

• What it graded. DeepSeek V4.1 Flash (deepseek-v4.1-flash-1017), run on 2026-11-09 on DigitalOcean’s serverless inference, graded 1,168 AACR-Bench reviews by seven reviewers: QuickReview’s frozen build and the six third-party reviewers of the paper’s final comparison (the pi-review panel, Claude Code, open-code-review, Codex CLI, OpenCode and PR-Agent). These are all the reviews of those seven that the GPT-6 Sol judge was asked about. The dataset files this judge under second_judges/, and the benchmark paper calls it the third judge, after the Opus judge. This is a different set from the 1,295 reviews of the earlier study that the Opus 5.5 judge graded (note 3); only Claude Code’s run is in both.
• Its handicap. It received the GPT-6 Sol judge’s prompt after one added system line saying it had no tools and no repository, so it judged from the message alone: the defects with their quoted evidence, the rejected claims and the review. The GPT-6 Sol judge ran as an agent with the repository checked out and made about 6 tool calls per judgment. The DeepSeek judge ran at high effort, and at low effort where an answer at high was cut off (1,056 and 112 of the 1,168 accepted answers).
• Per defect. The dataset card and the paper give the agreement per reviewer: the same verdict on 97.9% to 99.6% of each reviewer’s pairs, with kappa from 0.93 to 0.99. Pooled over the seven reviewers, counted for this post from the dataset’s tables, that is 3,601 of 3,640 reviewer-defect pairs (98.9%, kappa 0.978) and 624 of 630 high-severity pairs. Of the 39 disagreements, 37 involve a partial on one side and 2 are found against missed. With the GPT-6 Sol judge’s verdict first and the DeepSeek judge’s second: found and found 2,136, found and partial 16, found and missed 1; partial and found 8, partial and partial 33, partial and missed 5; missed and found 1, missed and partial 8, missed and missed 1,432. Recall moves by at most 0.4 points for any reviewer, and the order of the seven is the same on recall, on high-severity recall and on recall weighted by severity. The frozen build finds 89 of the 90 high-severity defects under this judge, against 87.5.
• Per comment. The two judges give the same five-way grade on 84.0% to 92.1% of each reviewer’s comments. Comments correct is 1.2 to 9.3 points lower under the DeepSeek judge for six of the seven reviewers. Most of those disagreements grade as incorrect a comment that the judge with access to the code graded correct or partly correct.
• Two other benchmarks with answer keys. There the DeepSeek judge ran at low effort through the same bridge as the GPT-6.1 Sol judge that scores them. Against that judge, counted from the dataset’s tables, it gave the same matched-or-not verdict on 1,172 of 1,211 golden comments on Martian’s 50 PRs (96.8%, seven tools) and on 4,581 of 4,640 issues on Qodo’s PR-Review-Bench (98.7%, eight tools). It is the more lenient judge on both: Martian’s F1 is 0.9 to 3.8 points higher for every tool and Qodo’s F1 is 0 to 1.4 points higher. Where two tools were already close, the order changed (note 18).
• Status. No human has checked this judge either. Its numbers show how much the published results depend on the judge, and they do not correct them. Its run cost $8.96 for AACR-Bench by the operator’s record, which the published files cannot confirm, against $41.85 for the GPT-6 Sol judge on the earlier study’s 1,295 reviews.

18. Where the scores bunch up

• Martian, 20 PRs, the three leaders. QuickReview on Sonnet 5.5 (F2 43.9), Claude Code high on Opus 5.5 (43.2) and Claude Code high on Sonnet 5.5 (43.1) meet in 33 of the 298 matchups. QuickReview on Sonnet 5.5 against high on Opus 5.5: 20 matchups, 10 wins each under the GPT judge and 5 against 15 under the Opus judge, the same winner in 13. QuickReview on Sonnet 5.5 against high on Sonnet 5.5: 5 matchups, 5 against 0 under the GPT judge and 1 against 4 under the Opus judge, the same winner in 1. high on Opus 5.5 against high on Sonnet 5.5: 8 matchups, 5 against 3 under both judges, the same winner in 4. In all, the same winner in 18 of 33; wins of 15, 15 and 3 under the GPT judge and 6, 20 and 7 under the Opus judge, of 25, 28 and 13 matchups. high on Sonnet 5.5 was not in the final round, so its counts are the smallest.
• Martian, 50 PRs, the two leaders. On Martian’s Core F2, among the seven tools scored (QuickReview’s frozen build, three open tools, and Martian’s published reviews of qodo-extended-v2, cubic-v2 and claude-code, re-judged), qodo-extended-v2 scores 59.3 and cubic-v2 58.9 under the GPT-6.1 Sol judge, as in the repository’s BENCHMARKS.md, and 60.7 and 61.1 under the DeepSeek judge, as in the README of its second_judges folder. Lower down, claude-code (42.8) passes QuickReview (41.9) under the DeepSeek judge, against 40.1 and 40.9.
• AACR-Bench, high severity. The frozen build finds 87.5 of the 90 high-severity defects (97.2%), Codex CLI 81 (90.0%) and the pi-review panel 80 (88.9%), under the GPT-6 Sol judge. The paper puts the frozen build ahead of the panel by 8.3 points (95% interval 1.9 to 14.6) and of Codex CLI by 7.2 (0.6 to 14.1). Among the compared build of 1 October (83.5), Codex CLI, the panel and QuickReview’s low preset (79.5) it establishes no lead, and says that with 90 high-severity defects, gaps of a few points are below what the benchmark can resolve. Over all 520 defects the frozen build finds 461 (89%), the panel 347.5 (67%) and Codex CLI 287 (55%).
• How far “saturated” goes. Only the high-severity list is near its maximum. The two Martian cases bunch at about 43 and 59 of 100, where the key is short (note 10). The full AACR-Bench key separates the final comparison’s reviewers from 37% to 89%.