AI Code Review Benchmark: Sonnet vs Opus on 30 Real Bugs

In the previous article I moved my commit review from Opus to Sonnet and cut the cost by 62%. One question stayed open: how many bugs get lost in the switch? Without a reference set there is no answer, so I built an AI code review benchmark from the history of my own repositories: 18 commits containing 30 bugs that were later fixed by hand. I ran the same review script on them in two modes.

Sonnet found 18 of the 30 bugs for $11.68. Opus found 17 for $26.03.

The short version:

1. Opus did not find more on this set. One bug of difference out of 30 means nothing, but there is nothing yet to justify paying double.
2. The models missed different bugs. Together they found 21, while each alone found 17 or 18.
3. Two of the nine bugs that both models missed are caught by tsc, for free and in seconds.
4. The other misses look alike: the bug is only visible together with a file that is not in the diff, or with knowledge of how a library behaves at runtime.
5. Half of the candidates the automatic search produced were not bugs. Without manual labeling, the reference set lies.

Where to get bugs with a known answer#

AI review vendors grade themselves. DeepSource said it plainly: every vendor runs a benchmark on its own dataset and wins on it. I needed an answer about my scripts and my code.

The source is already in git. Every commit with fix in its message changes lines that someone once wrote. Running git blame on those lines as they were before the fix shows the commit that introduced the bug. The technique is called SZZ, after the authors of a 2005 paper: Śliwerski, Zimmermann, Zeller.

git show -U0 --format= "$FIX" -- "$FILE" | grep '^@@'
git blame --porcelain -L 41,41 "$FIX^" -- "$FILE" | head -1

A Python script went through seven repositories: two Next.js sites, a catalog with a Prisma database, a web calculator, an open-source TypeScript wiki, an iOS app and an Apple Watch app in Swift. It found 125 "bug commit → fix" pairs.

Then came the manual work. I dropped fixes to data and copy and kept 37 candidates with code changes. A separate Sonnet agent labeled each one: it read the fix and the original commit and answered three questions. Is this a real defect rather than a new requirement? Is it present in this commit? Could an attentive reviewer have spotted it without running the app on a device? I rechecked the multi-bug cases myself.

Of the 37 labeled, 18 remained. The filtering turned out more useful than I expected:

• some "fixes" implemented a new requirement, such as syncing the theme between devices;
• in some, the bug came from a different commit, and SZZ pointed at a neighbor that merely touched the same lines;
• in some, the defect only showed up later, on data that did not exist when the commit was made.

One more detail, without which the numbers are worthless. If you give a reviewer a working copy with full history, git log --all shows the future fix together with the bug description. So I built each case as a separate repository of two commits: the state before the bug and the bug commit itself. It contains no future.

How I ran it#

The review was done by the same change-review that runs for me every day. I gave it three arguments: where the repository is, which files to limit the review to, and not to write state. There are two modes:

After each review, a separate judge agent compared the findings with the reference: same cause and same failure, the line number may differ. For findings that matched nothing in the reference, the judge read the code and marked each one as a real defect or not.

AI code review benchmark results#

I calculated prices from message.usage in the agent logs, as in the previous article. The judge cost another $1.63 and $1.70, and labeling the reference set cost $3.94. The whole experiment came to about $45.

Of the 14 serious bugs, each model found nine, but not the same nine. Opus caught a paywall that promises a trial to a subscriber who has already used one. For that you need to know that in StoreKit, eligibility for an introductory offer is checked separately from the offer itself. Sonnet caught a first-run setup of the wiki that any visitor to a fresh server could complete and become the administrator. Sonnet has four bugs that Opus lacks, and Opus has three that Sonnet lacks.

The two runs with different models together found 21 bugs, more than either alone. They cost $37.71, more than Opus alone. Running Sonnet twice would be cheaper, about $23, but I did not run that. There is a caveat too: I ran each mode once. Part of the difference may be run-to-run variance of the same model, and I have not measured that yet.

What nobody found#

Both models missed nine bugs. They fall into three groups.

Type errors. In the calculator, a function is called without the required weight parameter; in another file, a type IntlMessages is used that is not declared anywhere. The build failed on both commits. In the same files the reviewers found 6 to 10 other problems each, yet they missed exactly these two. tsc --noEmit finds this without a single token. Hence the order: compiler, tests and linter first, and only then the model.

The bug lives at the seam with another file. Three cases:

• 'unsafe-eval' was removed from the CSP, but MDX runs on the client through new Function(), so articles break in production;
• serialize() compiles MDX in dev mode for a different JSX runtime than the one that later renders it;
• the sitemap is prerendered while the Docker image is built, where there is no database, and it serves an empty list for six hours.

Each time the diff contains one file, and on its own it looks fine. To see the bug you have to open the component that uses this config, or know where the code runs. A reviewer is allowed to read the surrounding code but, on a limited diff, does not. It is the same blind spot I wrote about last time: an incremental pass over one file does not see links between files.

Small stuff. Three low-severity bugs that are open to debate, and one exception for slug === "test" whose meaning is explained only in the message of the same commit.

What one evaluation costs#

Both runs together consumed about 64 million tokens. 82 to 85% of them were cache reads, which cost twenty times less than regular input. The mechanics are the same as in the previous article. Each agent starts with a context of about 49 thousand tokens, and its requests reread it again and again. For 18 cases that is about 130 agents per mode.

A large part of those 49 thousand has nothing to do with the task: the shared rules from CLAUDE.md and the memory of the project I launched the evaluation from. The agents reviewed other repositories but carried my site's rules with them. That is why one review even found, in an old commit, an assistant signature that my own rules forbid.

What I will change#

• The default mode stays on Sonnet. There is no data in favor of Opus for review. thorough stays for releases, as a second, independent pass after the default one.
• tsc and the build go before the review. Two of the nine misses are closed for free.
• Give the reviewer the files that use the changed code. Right now it gets only the diff. The list of callers can be found with grep before the model starts.
• The reference set now lives next to the scripts. Any change to the review prompt can be run on the same 18 cases to see whether it got better or worse.

What I did not check#

• Spread between runs. I ran each mode once. To say "the models miss different things" rather than "one run missed different things", you need at least three runs per mode.
• Sonnet labeled the reference set. I rechecked the multi-bug cases but not all of them. If the labeler missed a bug in a commit, it never made it into the reference, and the reviewer's finding went into "extra".
• The judge is also Sonnet, at a low effort level. I did not check by hand its verdict on the extra findings (57 real out of 77).
• 30 bugs is few. A difference of fewer than 3 to 4 bugs between modes on a set like this is noise.

If you have your own review scripts, start with one evening: SZZ over the history, manual labeling, isolated cases. Without that, any "the new model reviews better" is an opinion, not a measurement. How the workflow scripts themselves work, I covered in the article on Claude Code workflows. Where their token usage comes from, I covered in the article on Claude Code token usage.