TL;DR
I ran QuickReview 1.6.0 on GPT-6.2 Sol over all 185 pull requests of AACR-Bench and asked two AI judges, GPT-6.1 Sol and DeepSeek V4.1 Flash, which review of each pull request was better: the new one or the one from QuickReview’s October build on GPT-6.1 Sol. Each judge saw every pair in both orders. Both judges agree that the new reviews find more of the defects that matter (+115 Elo pooled) and lose on wasted attention, finding quality and writing. On the overall verdict they split: the GPT judge leans to the new setup by 37 Elo, the DeepSeek judge to the old one by 77. Weighted equally they come out at -17 Elo, an interval that spans zero. The new setup left 8.1 comments per pull request against 6.9, cost $3.25 against $2.06, and took a median 11.0 minutes against 17.5. Its defect judge called 2.5 of its comments per pull request real findings outside the answer key, against 0.8, and its wrong comments nearly doubled, 0.95 per pull request against 0.52. On Martian’s and Qodo’s benchmarks, where one GPT-6.1 Sol judge scored both setups, the new setup matched fewer answer-key issues: 98 of Martian’s 158 against 104, with 128 more claims that match nothing in the key, and 35.2% of Qodo’s 580 issues against 45.3%. On Qodo’s it found functional bugs at the same rate, 53.1% against 52.1%, and the whole drop is in rule violations, 14.8% against 37.6%. Most of the October build’s rule-violation matches came from nits, which the quiet default does not post.
OpenAI’s GPT-6.2 Sol has the same list price as GPT-6.1 Sol ($2 per million input tokens, $10 per million output), so a team running QuickReview on Sol can switch at no change in price per token. I wanted to know whether it should. The answer-key benchmark I usually report no longer separates the two well: on the 90 high-severity defects of AACR-Bench both setups miss two or three. So, as in an earlier post on AI judges, I asked judges to read both reviews of the same pull request and pick the better one. This time I used two judges from different model families, ran every pair in both orders, and pooled the verdicts.
The setup
| New setup | October setup | |
|---|---|---|
| Reviewer | QuickReview 1.6.0 (19 November 2026) | QuickReview’s build frozen on 4 October 2026 |
| Model | GPT-6.2 Sol, medium reasoning (high for the deep tier) | GPT-6.1 Sol, the same reasoning levels |
| Runtime | pi built from its unreleased main branch (it reports 1.5.2), with the SDK engine | pi 1.0.0, from before the SDK engine |
| Quiet default | On: no nits, nothing below P2 | Not in that build |
| Pull requests | All 185 of AACR-Bench | The same 185, the published run |
Four things changed between the two setups: the model, about six weeks of QuickReview releases, the runtime under QuickReview (pi and the SDK engine), and the quiet default that 1.6.0 turns on. The new run’s pi was built from its unreleased main branch for GPT-6.2 Sol, so a released install today does not match it exactly. This compares what a user got in October with a current QuickReview build on the newer model, and it cannot say how much of a difference comes from the model alone. Both runs went through a ChatGPT login, so nothing was billed per token, and every cost below is a list-price equivalent.
On the answer key
The answer key is the verified list of 520 defects on the 185 pull requests that the reviews are scored against, 90 of them high severity.
| Answer-key defects found | New setup (GPT-6.2 Sol) | October setup (GPT-6.1 Sol) |
|---|---|---|
| High severity (of 90) | 87 | 87.5 |
| Medium (of 145) | 135 | 134.5 |
| Low (of 285) | 213.5 | 239 |
| All (of 520) | 435.5 | 461 |
A defect counts half when a comment finds it in part. On high and medium severity the two setups are level. The new setup finds 25.5 fewer low-severity defects, which is what the quiet default is for: it drops nits and anything below P2 before posting. The defect judge was GPT-6.1 Sol at high reasoning for the new run and GPT-6 Sol at high reasoning for the published October run. In the earlier post, defect judges from three model families agreed on about 99% of these checks.
The quiet default did not make the new setup quieter. Its defect judge read 1,501 of its comments (1,498 inline and 3 in the review body), 8.1 a pull request, against 1,278 (6.9) for the October build. The comments on an answer-key defect are about the same, 499 against 507. The extra ones are all off the key, where the new setup left 832 correct comments against 688, and 170 that the defect judge found incorrect, partly correct or unverifiable against 83. Counting every comment, 176 (12%) against 97 (8%) were not correct.
The new setup cost $3.25 a pull request against $2.06, with every model call of the review priced at list. The price per token is the same, so the difference is token use: per review, the new setup read about 2.1 times as many tokens and wrote about 1.9 times as many. It was also faster, a median 11.0 minutes per review against 17.5.
Two judges, every pair in both orders
Each judgment is one model call with no tools. The judge gets the pull request’s title and description, its diff, the files it touches at the head commit, the answer-key defects for that pull request, and the two reviews, rewritten into one neutral format and labelled A and B. It is told that the answer key is a strong reference and not ground truth, so a finding outside the key can count as a real defect when the judge can point at the code. It picks a winner, or a tie, in four categories and overall:
- Finds the defects that matter: the answer-key defects a review raises plus real defects the key lacks, weighted by severity.
- Wastes less attention: fewer findings that are wrong, unreachable or not worth raising.
- States findings better: correct severity, precise location, a concrete failure, the root cause, a usable fix.
- Easier to read: clear, concise, organised.
- Overall: which review a maintainer of the repository would rather have received.
The judges were GPT-6.1 Sol and DeepSeek V4.1 Flash, both at xhigh reasoning effort. GPT-6.1 Sol judged each of the 185 pairs twice, once with the new review as A and once as B. DeepSeek, which is much cheaper, judged each pair up to four times, twice in each order. DeepSeek did not return a verdict on 5 pull requests: on those it was still reasoning when the 10-minute limit for a call ran out, in every attempt.
Running both orders matters more than I expected. GPT-6.1 Sol picked Review B in 57% of its decided verdicts, where a judge without a preference for a position would be near half. DeepSeek picked Review A in 51%. With each pair judged once in each order, GPT’s preference falls on both setups equally. The order also changed GPT’s mind on some pairs: of the 185 it judged both ways, the same setup won both times on 149, the winner flipped with the order on 25, and 11 had a tie in one order.
The verdicts become ratings with a Bradley-Terry fit on the Elo scale, ties counting half, and 95% intervals from a bootstrap over pull requests. A gap of 100 Elo means the higher setup is expected to win 64% of pairs.
Where the judges agree
In all four categories the two judges point the same way, and every pooled interval excludes zero.
| Category | GPT-6.1 Sol judge | DeepSeek V4.1 Flash judge | Pooled |
|---|---|---|---|
| Finds the defects that matter | +163 | +68 | +115 (+78 to +152) |
| Wastes less attention | -64 | -145 | -101 (-139 to -64) |
| States findings better | -86 | -68 | -78 (-111 to -48) |
| Easier to read | -98 | -113 | -105 (-142 to -69) |
| Overall | +37 | -77 | -17 (-53 to +18) |
Elo gap of the GPT-6.2 Sol setup over the GPT-6.1 Sol setup; positive favours GPT-6.2 Sol. Pooled intervals are 95%.
The new reviews find more. That holds although they find fewer answer-key defects, because the judges also credit real defects outside the key, and the new setup posts more of those. They are also longer, carry more findings the judges consider wrong or not worth raising, and state their findings less well.
A langflow pull request that changes a parser component shows the trade. In both orders the GPT judge gave the new review the category for finding more and the overall verdict to the October one, as it did on 23 of the 185 pull requests. Both reviews raised the main template failure and three other answer-key defects, though the October review called one of them, list inputs rejected before conversion, an internal mismatch. The new review also caught whitespace-only rows that survive cleaning, a key defect the October review missed, and the October review got half credit on one the new review missed. The GPT judge counted the new review’s extra findings, among them keyword expansion that fails on non-string column labels, which the defect judge graded incorrect, as “meaningful advantages, not merely a larger finding count”. The new review also declared eleven blockers, many of which depended on editor, graph or deployment behaviour the code did not show, and the judge preferred the review that asked less of the maintainer.
Where they split
Asked which review a maintainer would rather receive, the judges weigh those differences in opposite ways. The GPT-6.1 Sol judge puts the new setup 37 Elo ahead (-9 to +82); the DeepSeek judge puts it 77 Elo behind (-122 to -34). The GPT judge’s interval spans zero and DeepSeek’s does not. On the same pair in the same order, the two judges named the same overall winner 219 times out of 354, and opposite winners 120 times.
The earlier post found the same pattern with a GPT judge and a Claude judge: they agreed on which review found more and which wasted less time, and differed on how much each of those matters. My prompt says that catching serious defects and not wasting the author’s time matter most, and does not say which of the two wins when they conflict. Each judge settles that in its own way.
The GPT-6.1 Sol judge is from the same family as both reviewers, and it preferred the newer model’s reviews. DeepSeek is from neither, and it preferred the older ones. My data cannot tell whether that is a family preference or a difference in taste.
Pooling the two judges
To get one number I fit both judges’ verdicts together, weighted so that each judge counts about the same however many times it was asked: GPT’s 370 verdicts each count twice and DeepSeek’s 680 once. The pooled overall gap is -17 Elo (-53 to +18). The interval spans zero, so the pooled verdict cannot tell the two setups apart.
The pooled number is only useful next to the two it came from. Near zero, it could mean two judges who each found the setups level or two judges who disagree, and here it is the second.
The judging cost about $87 at GPT-6.1 Sol’s API prices for 370 calls (it went through a ChatGPT login, so it used quota, not dollars) and about $29 on DeepSeek V4.1 Flash, priced from the token counts of the 680 calls that returned a verdict. Another 311 calls timed out or returned no valid verdict, and report no token counts.
What the two setups posted
The ratings say which review the judges preferred. To see what they were weighing, I went through the comments themselves: the defect judge’s verdict on each of the 2,779 comments it read, the answer-key credit for each defect, and the reasons the two pairwise judges wrote. No judge was rerun for this part. The theme counts come from labels that Claude, an AI assistant, gave to small fixed random samples by reading each item, so they show the rough shape and their rates are approximate.
One caution applies to every correct and not-correct count below. The two runs had different defect judges, GPT-6.1 Sol for the new run and GPT-6 Sol for the October one, so these counts mix a difference in setup with a difference in judge. One visible sign is in the answer key’s list of claims that were checked and rejected. The GPT-6.1 Sol judge cites that list in 52 of the 170 off-key comments it found not correct for the new setup, while the GPT-6 Sol judge cites it in 2 of 83 for the October build.
Volume and tone
| New setup (GPT-6.2 Sol) | October setup (GPT-6.1 Sol) | |
|---|---|---|
| Comments per pull request, mean / median | 8.11 / 6 | 6.91 / 5 |
| Words per comment, median | 66 | 61 |
| Tagged blocker | 788 (52.5%) | 408 (31.9%) |
| Tagged question | 713 | 407 |
| Tagged nit | 0 | 463 |
| Priority P0 / P1 / P2 / P3 | 5 / 451 / 1,042 / 0 | 0 / 236 / 422 / 587 |
| Review verdict: changes requested / approve | 156 / 29 | 140 / 45 |
Priorities are counted on the posted comments; 3 and 21 carry none. The October build posted 1,266 comments, and its defect judge saw 12 of them twice, which gives the 1,278 that the tags and the rest of this section count.
The quiet default shows in the tags. The new setup posted no nits and nothing at P3, while P3 is the October build’s largest priority group. It also tagged more than half of its comments as blockers, against about a third. It posted more comments than the October build on 85 of the 185 pull requests, fewer on 59 and the same number on 41.
The answer key, defect by defect
The totals in the answer-key table above hide how much the two setups overlap. Here a defect counts as caught by a setup when that setup gets any credit for it:
| Severity | Defects | Caught by both | New setup only | October setup only | Neither |
|---|---|---|---|---|---|
| High | 90 | 87 | 0 | 2 | 1 |
| Medium | 145 | 131 | 4 | 5 | 5 |
| Low | 285 | 192 | 24 | 48 | 21 |
| All | 520 | 410 | 28 | 55 | 27 |
Thirteen defects have a half credit on at least one side. Split exactly by credit, the two setups share 407 of their points. The new setup’s own extra is 28.5 and the October build’s is 54, and 23.5 and 49 of those are low severity. The whole 25.5-defect gap on the key is in low severity. On high and medium severity the two setups find almost the same set of defects, and their own extras differ by half a defect in each.
Comments outside the key
The new setup left 1,002 comments on nothing in the answer key, the October build 771. The defect judge sorted each one:
| Off-key comments | New setup | October setup |
|---|---|---|
| Correct and worth raising | 461 (46%) | 154 (20%) |
| Correct but minor | 371 (37%) | 534 (69%) |
| Not correct | 170 (17%) | 83 (11%) |
Most of the new setup’s extra volume is real by its defect judge’s reading. Per pull request it adds 2.49 off-key comments the judge called correct and worth raising, against 0.83, and the judge flagged every one of them as a real defect the key lacks. The October build’s off-key material is mostly correct but minor, 2.89 a pull request against 2.01, and 394 of its 771 off-key comments are nits. The wrong comments grow too: across all comments, 0.95 a pull request for the new setup against 0.52.
To see what the wrong ones are, Claude read a fixed random sample of the off-key comments the judge found not correct, 30 of the new setup’s 170 and 20 of the October build’s 83, each with the judge’s reason, and gave each one theme:
| Why the comment was not correct | New setup (of 30) | October setup (of 20) |
|---|---|---|
| The behaviour predates the pull request | 9 | 4 |
| Misread the code: wrong mechanism, already handled, or cannot happen | 7 | 8 |
| A real code path with a contrived trigger | 6 | 7 |
| Argues with intended behaviour or goes beyond the pull request’s scope | 6 | 1 |
| Cannot be checked from the repository | 2 | 0 |
Half of the new setup’s sampled wrong comments (15 of 30) are about something real that the pull request did not cause or did not set out to do. The October build’s lean towards plain misreadings (8 of 20). A keyword count over all of them points the same way: the judge’s reason mentions pre-existing behaviour for 46 of the new setup’s 170 and 11 of the October build’s 83. Seven pull requests appear in both samples, and on several of them the two setups made the same wrong claim, so some of these mistakes repeat whichever model runs the review.
Why the judges pick each setup
On the overall question, GPT-6.1 Sol gave the new setup 194 wins, 21 ties and 155 losses, and DeepSeek gave it 257, 18 and 405. Both judges were asked for a reason with each verdict. From a fixed random sample of 15 explained games won by each setup from each judge, Claude labelled every reason the judge gave. A verdict can give several reasons, and reasons that neither judge gave more than twice are left out.
| Reason the judge gives | GPT-6.1 Sol judge | DeepSeek judge |
|---|---|---|
| New setup wins (of 15) | ||
| It catches a real defect the October review misses | 10 | 8 |
| The October review spends attention on low-value nits or test and doc requests | 4 | 4 |
| October setup wins (of 15) | ||
| The new review makes wrong, speculative or unestablished claims | 8 | 8 |
| The new review over-states severity (minor items and test requests as blockers) | 7 | 6 |
| The new review splits one root cause into several comments | 3 | 4 |
| It catches a real defect the new review misses | 2 | 9 |
| The judge says the new review found more, and picks the October one anyway | 8 | 4 |
The two judges see the same strengths and weaknesses. The new setup wins on an extra find and loses when it is noisier and louder. They differ in how much each counts. GPT decides 10 of its sampled wins for the new setup on an extra find, and in 8 of its wins for the October build it says the new review had the better recall. DeepSeek cites a find only the October review made in 9 of its wins for it, often a small one such as a misleading log message, a stale note or a test request.
DeepSeek left 260 of its 680 verdicts (38%) without reasons. The explanation is empty, ... or placeholder, and the category reasons are almost all empty too. The winners in those verdicts split like the explained ones: the October build takes 60% of the decided games among them and 62% among the rest, so they do not tilt the rating. The reasons above come from explained verdicts only, and GPT explained every verdict. Both judges also read the diff and the touched files rather than the whole repository, and GPT often discounts a claim because the supplied code does not establish it, so a true cross-file claim can read to them as unsupported.
Where they pick opposite winners
Taking each judge’s majority over its games on a pull request, on the 180 pull requests both judged the judges pick the same winner on 89 (the October build 48, the new setup 41), opposite winners on 40, and one judge is even on 51. On 32 of the 40 opposite cases, GPT picks the new setup and DeepSeek the October build.
| Pull requests | Count | New setup: comments / not correct, per pull request | October setup: comments / not correct, per pull request |
|---|---|---|---|
| Both judges pick the October build | 48 | 9.77 / 1.54 | 7.02 / 0.73 |
| Both judges pick the new setup | 41 | 6.98 / 0.88 | 6.71 / 0.49 |
| GPT picks the new setup, DeepSeek the October build | 32 | 7.72 / 0.50 | 7.72 / 0.47 |
When both judges pick the October build, the new review is the noisy one, with 9.77 comments and 1.54 wrong ones a pull request against 7.02 and 0.73. On the 32 split pull requests the two reviews look alike by these counts, and the October build is slightly ahead on answer-key credit, 2.69 against 2.34 a pull request. There the judges even split on which review found more: GPT gives that category to the new setup 38 to 3, DeepSeek to the October build 75 to 32. They disagree on which findings are real and which count, as well as on how much noise to tolerate.
Claude also read every game on four of these pull requests (in aspnetcore, FreeCAD, gemini-cli and timescaledb). Each time, GPT’s deciding reason was a finding only the new review made outside the answer key, such as a migration risk, or the new review being leaner. On three of the four, DeepSeek’s was the October review covering more answer-key items or small real ones, such as a stale comment, a copy-paste slip or an unused import. On the fourth, DeepSeek picked the new review in one order and the October one in three, and those three came without reasons.
Four pull requests
Defects that only the October build caught outnumber those that only the new setup caught, 55 to 28, and on the pull requests where both judges pick the same winner the October build takes 48 and the new setup 41. The first two examples go the new setup’s way and the last two the October build’s.
three.js, “Automatically clean shape data passed to ExtrudeGeometry”. The pull request welds together shape points closer than an absolute 0.0002 units, and one medium-severity defect in the key is that this threshold damages valid small shapes. The new review raised it as a question and got the credit: “A rectangular notch 0.0001 units wide and 0.75 units deep loses a bottom corner, filling half of the cutout.” The October review left a nit that the judge graded correct but minor: “Please document that 0.0002 is measured in shape coordinates and explain the rationale for this cutoff, including its effect on small features.” Both caught the other two key defects, one of them high severity. The new setup won all six games, two from GPT and four from DeepSeek. GPT: “it catches an important class of geometry corruption that [the October review] largely reduces to a documentation nit.”
Elasticsearch, “Add ability to set max_analyzed_offset implicitly to index.highlight.max_analyzed_offset”. The key has a medium-severity defect: nothing gates the newly accepted value -1 by transport version, so older nodes in a mixed cluster reject it. The new review found it: “During a rolling upgrade, the upgraded coordinator accepts -1 here and writes it unchanged (AbstractHighlighterBuilder.java:212). An older shard node feeds it through the previous positive-only setter on deserialization, so the shard fails before highlighting”. The October review said nothing about it, and its extra comment was a documentation nit. The new setup won all six games. GPT: “it identifies a real rolling-upgrade failure that [the October review] misses.”
browser-use, “Add anti bot detection via patchright”. The pull request hard-codes channel='chrome' in the default browser launch. The key’s high-severity defect is that an install with only the documented Chromium then fails to start, and two medium ones are that the same argument reaches the Firefox and WebKit launchers and that headed contexts lose their window-sized viewport. The October review raised all three as hedged questions and got half credit on each: “The default launch now requests channel=‘chrome’, but README.md:31-34 only installs Chromium, and BrowserConfig offers no channel override or fallback.” The new review did not mention the launch channel, and its viewport comment was about headless contexts. Both caught the two low-severity defects in the key, and the new review spent two blockers on one of them. The October build won all six games. GPT: “receiving those warnings is more valuable than [the new review’s] more forceful but narrower security-focused review.”
typescript-go, “Port more of tsc help from strada”. The new review posted 15 comments, 14 of them blockers, all on the help-text renderer. Each said the new help advertises something this staged port has not implemented yet, for example “tsgo tsc --showConfig -p . loads the project, then returns ExitStatusNotImplemented without printing it”. The defect judge graded 13 of the 15 partly correct, mostly as pre-existing behaviour or beyond the port’s scope. The new review’s one answer-key find, a low-severity latent nil dereference, was its 15th comment, a question. The October review left one nit and approved: “Please remove the zero-valued terminalWidth variable and dead branch, flatten the implemented else body, and retain a concise TODO for terminal-width formatting.” The October build won 5 of the 6 games (GPT 2 of 2, DeepSeek 3 of 4). GPT: “the additional established issue is low severity while [the new review] imposes a disproportionate and poorly justified blocking review.”
Martian’s and Qodo’s benchmarks
The October build’s results on two other code review benchmarks are in an earlier post, so I ran the new setup on both and scored it the same way. Martian’s Code Review Bench, offline set, has 50 pull requests and 173 answer-key comments, 158 of them in its default scoring profile, which Martian calls Core. Martian’s own pipeline splits each review into claims and matches them against the key. Qodo’s PR-Review-Bench has 100 pull requests into which an LLM injected 580 issues: 309 functional bugs and 271 violations of rules the repository writes down. A comment finds an issue when the judge matches the two and the comment sits on the issue’s file within five lines of it.
Unlike the AACR-Bench defect judge above, one judge scored both setups here: GPT-6.1 Sol at low reasoning, through Martian’s pipeline and, for Qodo, which publishes no scorer, through the scorer the October build was scored with. The two scorings ran 47 days apart, with the same Qodo prompt and matching rule. Both builds carry changes designed from misses on 20 of Martian’s 50 pull requests and on 59 of Qodo’s issues, among them the checks of written repository rules (earlier post, note 6), so neither benchmark is fully held out for either setup. Both setups reviewed the same pull requests, and every review finished.
Martian
| Martian, Core profile | New setup (GPT-6.2 Sol) | October setup (GPT-6.1 Sol) |
|---|---|---|
| Answer-key comments found (of 158) | 98 | 104 |
| Claims that match nothing in the key | 664 | 536 |
| Precision | 12.9% | 16.2% |
| Recall | 62.0% | 65.8% |
| F1 | 21.3 | 26.1 |
| Inline comments per pull request | 15.0 | 12.4 |
| Cost per pull request | $5.40 | $2.84 |
The new setup found 6 fewer of the 158 and made 128 more claims that match nothing in the key, so precision, recall and F1 all fall. The gap is not at the top of Martian’s severity labels: across all 173 answer-key comments, both setups found 48 of the 66 labelled Critical or High, and the new setup found 55 at Medium and Low against 60. Of the 86 answer-key comments that an audit by AI agents in the earlier post judged to be real defects, it found 70 against 75.
The gap of 6 matches is larger than the judge moved on unchanged reviews, but not by a measured margin: the reviews of three tools that Martian publishes were re-judged, unchanged, in both scorings, and between the two their Core matches moved by up to 3 and their unmatched claims by up to 4. Those reviews make about a quarter as many claims as QuickReview’s, and each setup was scored once, so there is no interval. Martian also reads only the first 30 inline comments of a pull request, which left out 53 of the new setup’s comments on 4 pull requests and 12 of the October build’s on 2.
Qodo
| Qodo, 580 injected issues | New setup (GPT-6.2 Sol) | October setup (GPT-6.1 Sol) |
|---|---|---|
| Comments per pull request | 7.3 | 7.1 |
| Precision | 28.0% | 37.3% |
| Recall | 35.2% | 45.3% |
| F1 | 31.2 | 40.9 |
| Recall on functional bugs (of 309) | 53.1% (164) | 52.1% (161) |
| Recall on rule violations (of 271) | 14.8% (40) | 37.6% (102) |
| Cost per pull request | $3.55 | $2.48 |
The new setup’s review of one pull request posted no comments and counts as finding nothing.
On functional bugs the two setups are level, and the whole drop in recall is in rule violations. With about the same number of comments a pull request, the new setup found 164 functional bugs against 161 and 40 rule violations against 102. Issue by issue, 155 functional bugs were found by both setups, 9 only by the new one and 6 only by the October build. Of the rule violations, 38 were found by both, 2 only by the new setup and 64 only by the October build.
Most of the October build’s rule-violation matches came from comments the quiet default does not post. Of its 102, 71 came from comments at priority P3 or nit, and 70 from comments tagged nit; the new setup posted no nits. Counting only comments marked P0 to P2, the new setup matched more rule violations than the October build, 39 against 31, plus one from a finding in its review body, which carries no priority. That fits the quiet default causing the drop, but it does not measure it: the model, the build and the quiet setting changed at once, and no run has QuickReview 1.6.0 on GPT-6.2 Sol with quiet off. Before the checks of written rules were added, the 2 October build found 14.8% of the rule violations, the same share as the new setup (earlier post, note 6).
Each setup has one review per pull request and one judge pass on each benchmark, and Qodo’s scoring had no check of the judge’s own spread. The costs are list-price equivalents at the same price per token as on AACR-Bench, and separate from the AACR-Bench figures.
What it adds up to
On AACR-Bench, QuickReview 1.6.0 on GPT-6.2 Sol finds as many of the serious answer-key defects as the October build on GPT-6.1 Sol, and more real defects overall by both judges’ reading. It posts more comments, more of them wrong, and both judges find them harder to act on. Most of its extra comments are real findings outside the key by its defect judge’s reading, and its wrong ones lean towards problems the pull request did not cause or did not set out to solve. It costs about 1.6 times as much per review at the same price per token, and finishes sooner. Whether that trade is better overall depends on which judge you ask, and pooled they call it about even.
On Martian’s and Qodo’s benchmarks, with one judge for both setups, it matched fewer answer-key issues. On Martian’s it found 6 fewer of 158, more than the judge moved on unchanged reviews, though no interval was measured, and made more claims outside the key. On Qodo’s the functional bugs are level and the loss is in rule violations, most of which the October build had raised as nits that the quiet default no longer posts.
For a team on Sol, the cost, the comment volume and the rule violations are the measurable differences. A team that relies on QuickReview to check its written rules should know that the quiet default leaves out the low-priority comments that carried most of the October build’s rule-violation matches; whether turning it off (the review.quiet setting) brings them back on GPT-6.2 Sol is untested. The quality difference is not settled by these two judges, and the next step is people reading a sample of the pairs where the judges split.
QuickReview is open source at github.com/quickchatai/quickreview.
Data
The new run’s posted reviews, its defect verdicts and every pairwise judgment from both judges, with the per-category verdicts and the reasons, are in the QuickReview benchmark dataset on Hugging Face, in the gpt62_sol folder. The same folder has the new setup’s Martian and Qodo reviews with the judge’s verdicts on them. The October build’s are in the dataset’s Martian and Qodo tables, as the frozen build.
Frequently asked questions
Does QuickReview review better on GPT-6.2 Sol than on GPT-6.1 Sol?
Two AI judges do not agree. Over all 185 AACR-Bench pull requests, each judging every pair of reviews in both orders, a GPT-6.1 Sol judge put QuickReview 1.6.0 on GPT-6.2 Sol 37 Elo ahead of the October build on GPT-6.1 Sol (95% interval -9 to +82), and a DeepSeek V4.1 Flash judge put it 77 Elo behind (-122 to -34). With the two judges weighted equally the gap is -17 Elo (-53 to +18). The two setups also differ in QuickReview’s version, the pi runtime under it and the quiet default, so this is a comparison of setups, not of models alone.
Does QuickReview on GPT-6.2 Sol find more bugs?
On AACR-Bench, both judges say its reviews find more of the defects that matter, by +115 Elo pooled (+78 to +152), counting real defects outside the answer key. On the answer key itself it found 87 of the 90 high-severity defects against 87.5 for the October build, and 435.5 of all 520 against 461; the difference is in low-severity defects, which QuickReview 1.6.0’s quiet default leaves out on purpose. On two other benchmarks, scored by one GPT-6.1 Sol judge for both setups, it matched fewer answer-key issues: 98 of the 158 in Martian’s default profile against 104, and 35.2% of the 580 issues in Qodo’s benchmark against 45.3%. On Qodo’s it found functional bugs at the same rate, 53.1% against 52.1%, and the loss is in violations of repository rules, 14.8% against 37.6%.
What are the extra comments QuickReview leaves on GPT-6.2 Sol?
Mostly real findings outside the answer key. Its defect judge called 2.49 of its off-key comments per pull request correct and worth raising, against 0.83 for the October build, whose off-key comments are mostly correct but minor. Its wrong comments rose from 0.52 to 0.95 per pull request, and in a sample of 30 of the wrong ones outside the key, 15 were about something real that the pull request did not cause or did not set out to do. It posted no nits and tagged 52.5% of its comments as blockers, against 31.9%.
How much does QuickReview cost on GPT-6.2 Sol?
$3.25 per pull request at list prices on AACR-Bench, against $2.06 for the October build on GPT-6.1 Sol. GPT-6.2 Sol has the same list prices as GPT-6.1 Sol, so the difference is token use. A review took a median 11.0 minutes against 17.5.
How do you combine two AI judges into one rating?
Fit one Bradley-Terry model to both judges’ verdicts with each judge weighted equally, so that a judge asked more often does not count for more, and bootstrap over pull requests for the interval. Report each judge’s own rating beside the pooled one: a pooled number near zero can come from two judges pointing in opposite directions as well as from two who each call it even.