AI Code Review Benchmark 2026: Which AI Models Actually Catch Real Bugs
MacroscopeBench is an AI code review benchmark built on 12,000+ real bugs from 1,500+ public repositories across 14 languages. It asks whether an AI code reviewer would have caught each defect at the commit that introduced it, and scores every model on recall, precision, signal-to-noise, cost, and latency. Full leaderboard and methodology.
Almost every AI code review benchmark you can find in 2026 was published by a vendor who won it. That is not a coincidence, it is a design choice: pick the dataset, pick the judge, pick the metric, and the ranking follows. The harder problem is building an AI code review benchmark where the result can surprise you, and then publishing the leaderboard even when it does.
MacroscopeBench is that benchmark. It runs on real bugs rather than synthetic ones, and it asks a single question of every AI code reviewer: would you have caught this defect at the commit that introduced it? It is the same benchmark used to build and tune Macroscope Code Review, and the model leaderboard is public.
Short answer: MacroscopeBench evaluates AI code review on over 12,000 validated bugs drawn from more than 1,500 public repositories across 14 languages. Each model reviews a real commit diff without being told whether a bug exists, and two independent LLM judges score known-bug recall and overall precision. On the current all-bug leaderboard, GPT-6 Astra at max effort leads on score (78.0) with the highest precision (91.7%), GLM 5.3 at max effort has a near-identical score (77.4) at half the cost, and DeepSeek V4.1 Flash has the highest recall (83.1%) at $0.75 per review. No model leads on every axis, which is exactly why a production AI code reviewer routes across them.
TL;DR: What the AI Code Review Benchmark Shows
- Recall and precision pull in opposite directions. The model with the best known-bug recall (DeepSeek V4.1 Flash, 83.1%) has the worst precision of the leaders (63.5%). The model with the best precision (GPT-6 Astra, 91.7%) finds fewer of the known bugs (67.8%). An AI code review tool that optimizes only for recall buries your team in noise.
- Price and performance are barely correlated. GLM 5.3 at max effort scores 77.4 for $3.86 per review. GPT-6 Astra scores 78.0 for $7.87. A 0.6 point difference costs 2x.
- Signal-to-noise is the metric developers actually feel. GPT-6 Astra posts 6.26 useful comments for every noisy one. The bottom of the leaderboard sits near 1.1:1, which is one wasted comment for every real finding.
- Effort level matters more than model family. Claude Opus 5 moves from 59.4 at low effort to 69.9 at high effort, a bigger swing than the gap between several different vendors.
- The dataset is real production history, not puzzles. Over 12,000 validated bugs from over 1,500 public repositories in 14 languages, each anchored to the commit that introduced it.
- Every model is run three independent times. Reported metrics are the arithmetic mean of three runs, because a single run of a nondeterministic system is an anecdote.
- In our 2025 tool-level benchmark, Macroscope had the highest bug detection rate of the AI code review tools tested at 48.31%, ahead of CodeRabbit (45.76%), Cursor Bugbot (42.37%), Greptile (23.61%), and Graphite Diamond (18.26%).
The AI Code Review Benchmark Leaderboard
This is the current all-bug leaderboard from MacroscopeBench, showing the best score per model configuration. Score is the harmonic mean of known-bug recall and overall precision, so a model cannot climb it by shouting.
| Model | Effort | Score | Recall | Precision | Signal / noise | Cost | Duration |
|---|---|---|---|---|---|---|---|
| GPT-6 Astra | max | 78.0 | 67.8% | 91.7% | 6.26:1 | $7.87 | 3m 49s |
| GPT-5.6 Sol | high | 77.8 | 70.8% | 86.2% | 3.73:1 | $3.90 | 3m 0s |
| GLM 5.3 | max | 77.4 | 76.4% | 78.4% | 2.24:1 | $3.86 | 12m 30s |
| Grok 4.6 | xhigh | 75.9 | 69.7% | 83.3% | 3.67:1 | $4.68 | 14m 41s |
| Grok 4.6 | high | 75.1 | 69.9% | 81.1% | 3.17:1 | $3.51 | 15m 30s |
| Kimi K3 | max | 72.9 | 66.7% | 80.3% | 2.18:1 | $2.82 | 8m 13s |
| DeepSeek V4.1 Flash | max | 72.0 | 83.1% | 63.5% | 1.27:1 | $0.75 | 16m 17s |
| DeepSeek V4.1 Flash | low | 70.7 | 77.5% | 65.0% | 1.37:1 | $0.38 | 8m 9s |
| DeepSeek V4.1 Flash | high | 70.5 | 77.5% | 64.6% | 1.33:1 | $0.38 | 7m 40s |
| Claude Opus 5 | high | 69.9 | 76.6% | 64.2% | 1.20:1 | $6.50 | 4m 18s |
| Claude Opus 5 | medium | 65.5 | 68.8% | 62.5% | 1.18:1 | $3.50 | 2m 12s |
| Claude Opus 5 | low | 59.4 | 57.4% | 61.5% | 1.12:1 | $1.40 | 56s |
Best all-bug score per configuration. Score is the harmonic mean of known-bug recall and overall precision, averaged across three independent runs. The axis starts at 50 because every configuration scores above it.
Twelve configurations across seven models from OpenAI, Zhipu AI, xAI, Moonshot, DeepSeek, and Anthropic. The live leaderboard is filterable by severity bucket, which reranks the table: a model that is good at catching cosmetic issues is not necessarily the model you want guarding against data loss.
What This AI Code Review Benchmark Actually Measures
MacroscopeBench measures review, not repair. That distinction is the entire reason it exists.
Most AI coding benchmarks you have seen, SWE-bench chief among them, hand a model a known issue and ask it to produce a patch that passes tests. That is a repair task. It tells you whether a model can fix a bug that someone has already found, described, and filed.
AI code review is the opposite job. Nobody has found the bug yet. Nobody has filed anything. There is a diff, it may or may not contain a defect, and the reviewer has to decide, without a test suite pointing at the answer, whether to raise a flag. A model that tops SWE-bench can still be a poor AI code reviewer, because the skill being tested is not the skill being used.
So MacroscopeBench reconstructs the moment of review. For each task, the reviewer sees a repository diff and is not told whether a bug exists in it. Control commits with no known defect are mixed in, which is what makes precision measurable at all: a reviewer that flags something on every diff will be caught by the commits where there was nothing to flag.
The Dataset: 12,000+ Real Bugs From 1,500+ Repositories
The dataset is mined from production history rather than written for the benchmark. Over 12,000 validated bugs across more than 1,500 public repositories in 14 languages.
The construction logic is simple to state and laborious to execute. A bug enters the dataset when the project itself later identified and explicitly fixed it. That fix commit is the ground truth. The commit that introduced the defect is the task. The reviewer is placed at the introducing commit and asked to do its normal job.
This anchoring matters more than dataset size. A synthetic bug is a bug someone decided would be interesting. A mined bug is a bug that survived review, survived CI, shipped, and then cost someone real time. The distribution of what actually escapes into production looks nothing like the distribution of what a person would invent for a test set.
Every catalogued bug carries a severity grade:
| Severity | Definition |
|---|---|
| Critical | Permanent loss or compromise, such as unrecoverable data loss or unauthorized access |
| High | Serious disruption, such as crashes, hangs, or weakened data integrity |
| Medium | Degraded functionality or incorrect behavior in core paths without lasting damage |
| Low | Minor functional impact, or defects affecting rarely reached edge cases |
Filtering the leaderboard to Critical and High is the closest thing the benchmark has to a business question: of the defects that would have actually hurt, how many would this AI code reviewer have stopped?
How the Benchmark Scores AI Code Review
Four metrics, and the interesting one is precision. Here is what each means on MacroscopeBench.
Recall is the detection rate of the known bugs in the dataset. A bug is credited when an LLM judge determines a review comment matches a defect the project later explicitly fixed. Credit is granted at most once per bug per run, so a reviewer that describes the same defect five different ways gets counted once and recall can never exceed 100%. This closes the most common way a code review benchmark gets gamed.
Precision is the percentage of everything the reviewer reported, across the full benchmark, that a judge determines to be a valid bug introduced or exposed by the specific commit under review. The judge can explore the codebase when assessing validity. Invalid findings count against you. Subjective findings count against you. Out-of-scope findings, the pre-existing issues a reviewer noticed while passing through, count against you too, because on a real pull request they are noise.
Score is the harmonic mean of recall and precision. A harmonic mean punishes imbalance: you cannot post a 95 by being excellent at one and mediocre at the other.
Signal-to-noise is the ratio of useful comments to unuseful ones. Useful means a valid correctness issue at Medium severity or above. Unuseful means invalid for the task, or Low severity. This is the metric that most closely tracks whether developers keep reading the bot.
Two independent LLM judges evaluate results. Every model is run three independent times to account for nondeterminism, and every reported metric is the arithmetic mean of those three runs unless stated otherwise.
Why Precision Is the Metric Most AI Code Review Tools Hide
A high recall number with unreported precision is a marketing claim, not a benchmark result. Any reviewer can reach high recall by flagging more. The cost is paid by the developer, not the vendor.
Look at what the leaderboard does to that trade. DeepSeek V4.1 Flash at max effort finds 83.1% of known bugs, the best recall on the board, and its signal-to-noise is 1.27:1. For every four real findings it raises roughly three that are not. GPT-6 Astra finds 67.8% of known bugs and runs at 6.26:1. On a team reviewing a hundred pull requests a week, that difference is the difference between a bot people read and a bot people mute.
Every benchmarked configuration. Up and to the right is better. The point furthest right, the highest recall on the board, sits near the bottom of the precision axis, which is the whole argument against judging an AI code reviewer on detection rate alone.
This is why high-signal, low-noise is the right frame for judging an AI code review tool, and why the honest version of "which AI code reviewer is best" always has to name both numbers.
Cost and Latency: The Axes Benchmarks Usually Drop
MacroscopeBench reports dollars per review task and median wall-clock duration, because both are real constraints on GitHub PR review.
Cost is model inference cost in US dollars per code review task, with judging costs excluded. Duration is median wall-clock seconds per task, excluding judging and benchmark setup.
The spread is large and it does not track quality cleanly:
- $0.38 to $7.87 per review across the leaderboard, a 20x range.
- 56 seconds to 16 minutes 17 seconds median duration, a 17x range.
- The second-highest score on the board costs half of the highest, and the third-highest costs less than half.
- The fastest configuration is also the lowest-scoring one, which is the expected shape and a useful sanity check on the benchmark.
Latency is not a vanity metric on a pull request. A review that lands in three minutes is read before the author context-switches. A review that lands in sixteen minutes competes with whatever they opened next.
Why Most AI Code Review Benchmarks Cannot Be Trusted
The category has a selection-bias problem, and naming it is more useful than pretending otherwise. Public AI code review comparisons in 2025 and 2026 answer different questions with different ground truth and different scoring, and the publishing vendor wins a suspicious share of them.
The failure modes are consistent:
- Dataset chosen after the fact. Repositories and languages picked where the sponsoring tool is known to be strong.
- Recall reported without precision. Detection rate alone always flatters the noisiest reviewer.
- No control commits. Without diffs that contain no known bug, precision cannot be measured honestly.
- Multiple credits for one defect. Counting each comment rather than each bug inflates recall for verbose tools.
- A single run. LLM reviewers are nondeterministic, so one run per contender is noise dressed as a ranking.
- Synthetic or injected bugs. Defects written to be findable, in a distribution that does not resemble production escapes.
MacroscopeBench is a vendor benchmark too, and we will not pretend that disclosure makes it neutral. What we can do is publish the mechanics, run every model three times, credit each bug once, include control commits, score precision against an exploring judge, and put the leaderboard where anyone can read the results that do not flatter us. Our own preferred model does not lead every column on it.
MacroscopeBench vs Other Code Review Benchmarks
| Benchmark | What it measures | Ground truth | Reports precision |
|---|---|---|---|
| MacroscopeBench | Would a reviewer catch this bug at the introducing commit | 12,000+ mined bugs, 1,500+ repos, 14 languages, plus control commits | Yes, with control commits and an exploring judge |
| SWE-bench | Can a model patch a filed issue so tests pass | Real GitHub issues with test suites | Not applicable, it is a repair task |
| Vendor tool benchmarks | Detection rate of a chosen bug set | Varies, often undisclosed | Usually not |
| CVE and security benchmarks | Detection of known vulnerability classes | Published CVEs | Sometimes |
The useful takeaway is that these benchmarks are not competing answers to one question. SWE-bench tells you whether a model can fix. A CVE benchmark tells you whether it can spot a known vulnerability class. MacroscopeBench tells you whether it can review. If you are choosing an AI code reviewer for GitHub PR review, the third question is the one you are actually asking.
What the Benchmark Says About Choosing an AI Code Reviewer
No single model wins, which is the finding, not a hedge.
- If you want the fewest false alarms, the precision leaders (GPT-6 Astra at 91.7%, GPT-5.6 Sol at 86.2%) are the reviewers your team will keep reading.
- If you want maximum coverage on a critical release branch and you can absorb the noise, the recall leader (DeepSeek V4.1 Flash at 83.1%) finds more of what is there.
- If you want the best score per dollar, GLM 5.3 at max effort scores within 0.6 points of the leader at 49% of the cost.
- If latency is the binding constraint, the fast configurations trade several points of score for minutes of wall clock.
This is why Macroscope treats model choice as a routing decision rather than a brand loyalty. The benchmark is what makes the routing empirical: when a new model ships, it goes through MacroscopeBench before it goes near a customer's pull request, and it earns its place on the numbers or it does not get one.
How Macroscope Uses the Benchmark
Every change to Macroscope Code Review is validated against MacroscopeBench before it ships. The benchmark is not a marketing artifact that gets rebuilt once a year for a blog post, it is the regression suite for review quality.
That shows up in the product in a few concrete places:
- Model routing. Task complexity and severity sensitivity decide which model reviews, informed by per-severity benchmark performance rather than by whichever model is newest.
- Detection mode. Teams choose where they want to sit on the recall-precision curve, because the right answer differs between a payments service and an internal dashboard.
- Prompt and pipeline changes. A prompt edit that raises recall while quietly halving precision looks like an improvement in a demo and fails the benchmark.
- Check Run Agents. Custom checks defined in
.macroscope/*.mdinherit the same reviewer, so a team's own rules run on the same engine the benchmark measures. - Fix It For Me. Once a finding survives precision scoring, the AI code fixer can act on it with a commit or a PR rather than leaving a comment for a human to translate.
The 2025 Tool-Level Benchmark: Macroscope vs CodeRabbit vs Greptile
Before MacroscopeBench measured models, we ran the same discipline on tools. The 2025 benchmark assembled 118 self-contained runtime bugs from 45 popular open-source repositories across 8 languages, opened an isolated pull request per bug so no tool could free-ride on another's comment, and measured detection rate.
| AI code review tool | Bug detection rate | Bugs detected | Bugs evaluated |
|---|---|---|---|
| Macroscope | 48.31% | 57 | 118 |
| CodeRabbit | 45.76% | 54 | 118 |
| Cursor Bugbot | 42.37% | 50 | 118 |
| Greptile | 23.61% | 17 | 72 |
| Graphite Diamond | 18.26% | 21 | 115 |
Macroscope had the highest bug detection rate of the tools tested. The sample sizes differ because availability and rate limits differed: our access to Greptile's code review was disabled midway through evaluation, so its 72 is reported as 72 rather than quietly normalized to 118.
Full methodology, per-language breakdowns, and comment-volume data are in the 2025 AI code review benchmark writeup.
CodeRabbit Alternatives and Greptile Alternatives, Judged on Data
If you are evaluating CodeRabbit alternatives or Greptile alternatives, benchmark position is one input and pricing model is the other.
| Macroscope | CodeRabbit | Greptile | Graphite | |
|---|---|---|---|---|
| 2025 benchmark detection rate | 48.31% | 45.76% | 23.61% | 18.26% |
| Publishes a model-level benchmark | Yes, MacroscopeBench | No | No | No |
| Pricing unit | Usage: $0.05 per KB of diff | $24 per user per month billed annually (Essentials, $30 monthly), usage add-on on top | $30 per seat per month, 50 credits included, then $1 per extra credit | $40 per user per month (Team) |
| Free to start | $100 workspace credit, no card | Free tier with limits | Trial | Trial |
| Free for qualified open source | Yes | Varies | Varies | Varies |
Competitor pricing verified against each vendor's public pricing page as of August 2026.
The structural difference is the unit. Per-seat pricing made sense when the number of humans predicted the number of reviews. With coding agents opening pull requests, headcount and review volume have come apart, and usage-based pricing is the model that still tracks the work. More detail: best CodeRabbit alternatives and Greptile alternatives.
Limitations of This AI Code Review Benchmark
Stating these is part of the methodology, not a disclaimer at the bottom.
- Uniform prompting. Every model runs the same prompt, so per-model prompt optimization gains are not captured. A vendor tuning hard for one model would score differently.
- LLM judges, not human adjudication. Two independent judges are used, but they are models, and model judgment has failure modes.
- Severity distribution is designed for discrimination. The mix of severities is chosen to separate models, not to mirror the real-world frequency of critical versus low-severity defects. Do not read a recall number as a prediction of your own escape rate.
- Mined ground truth. A bug is in the dataset because the project found and fixed it. Defects nobody ever noticed are invisible to this method, as they are to every method.
- Public repositories only. Your private monorepo has conventions no public dataset contains, which is why the last section of this page is about testing on your own code.
How to Benchmark an AI Code Reviewer on Your Own Repository
Any public benchmark is an argument about method. Your repository is the evidence. The practical version takes an afternoon.
- Pick 20 merged pull requests that introduced a bug you later fixed. Your own revert commits and hotfixes are the dataset, and you already know the answer.
- Reopen each as a diff against its parent. Same shape as MacroscopeBench: the reviewer sees the introducing commit with no hint that something is wrong.
- Include 10 clean pull requests. Without them you are measuring recall only, which is the mistake the vendor benchmarks make.
- Count three things. Bugs found, valid findings, total comments. That gives you recall, precision, and signal-to-noise on your code.
- Score by severity, not by count. One caught data-loss bug outweighs a dozen style nits.
- Check the cost of the run. With usage-based pricing this is a real number on your invoice rather than an estimate.
The $100 new-workspace credit covers a trial of that size with room to spare, and there is no card required to start.
Getting Started With Macroscope
Macroscope installs on GitHub in a few minutes, reviews every pull request with the engine MacroscopeBench measures, and prices by diff reviewed rather than by headcount: $0.05 per KB of diff with a 10 KB minimum, roughly $0.95 for a typical pull request. CLI reviews run on Agent Credits at $0.01 each with 1,000 free every month. New workspaces get $100 of credit with no card, and Macroscope is free for qualified open source projects.
See the setup guide, the CLI for local reviews inside Claude Code, Codex, Cursor, or OpenCode, and the live benchmark for the numbers behind the reviewer.
Frequently Asked Questions
What is the best AI code review benchmark in 2026?
MacroscopeBench is the most complete public benchmark specific to AI code review, because it measures the review task rather than the repair task and reports precision alongside recall. It runs on over 12,000 validated bugs from more than 1,500 public repositories across 14 languages, mixes in control commits with no known defect so precision is measurable, credits each bug at most once per run, and runs every model three independent times. SWE-bench is the better-known benchmark but it measures whether a model can patch a filed issue, which is a different job.
Which AI model is best for code review?
It depends on which axis you are optimizing. On the current all-bug leaderboard, GPT-6 Astra at max effort has the highest score (78.0) and the highest precision (91.7%), GPT-5.6 Sol at high effort is within 0.2 points at half the cost, GLM 5.3 at max effort matches on score at $3.86 per review, and DeepSeek V4.1 Flash has the best known-bug recall at 83.1% for $0.75. No model leads every column, which is why a production AI code reviewer routes across models by task rather than standardizing on one.
What is the difference between recall and precision in AI code review?
Recall is the share of real bugs the reviewer found. Precision is the share of what it reported that was actually a valid bug introduced by the commit under review. They trade against each other: flagging more raises recall and lowers precision. A tool that publishes only its detection rate is showing you the easier half of the picture, since any reviewer can raise recall by commenting more aggressively.
How is MacroscopeBench different from SWE-bench?
SWE-bench gives a model a filed issue and a test suite and asks for a patch that makes the tests pass. MacroscopeBench gives a reviewer a commit diff, tells it nothing about whether a bug is present, and asks whether it raises the right flag. One measures repair, the other measures review. A model that ranks highly on SWE-bench is not automatically a good AI code reviewer, because finding an undisclosed defect and fixing a described one are different skills.
Can a vendor benchmark be trusted?
Treat the mechanics as the evidence rather than the ranking. The questions worth asking are whether control commits are included, whether precision is reported at all, whether each bug is credited once or once per comment, how many runs were averaged, and whether the dataset was fixed before the contenders were chosen. MacroscopeBench is published by Macroscope, and it publishes all of those, runs every model three times, and shows configurations where our preferred model is not on top.
What is a good signal-to-noise ratio for an AI code reviewer?
On MacroscopeBench, useful means a valid correctness issue at Medium severity or higher, and unuseful means invalid or Low severity. The leaderboard spans 6.26:1 at the top to about 1.1:1 at the bottom. Below roughly 2:1, most teams start treating the bot as background noise, which costs more than the bugs it catches, because a muted reviewer has a detection rate of zero regardless of its benchmark score.
How does Macroscope compare to CodeRabbit on bug detection?
In our 2025 tool-level benchmark across 118 real runtime bugs from 45 open-source repositories, Macroscope detected 48.31% and CodeRabbit detected 45.76%, both evaluated on the same 118 bugs. The larger practical difference is pricing model: CodeRabbit Essentials is $24 per user per month billed annually ($30 billed monthly) with a usage-based add-on on top, while Macroscope is $0.05 per KB of diff with no seat fee, which matters once agents are opening a meaningful share of your pull requests.
What are the best CodeRabbit alternatives based on benchmark data?
Macroscope is the strongest alternative on published benchmark data: highest detection rate in the 2025 tool benchmark, and the only one of the group that publishes a model-level code review benchmark. Cursor Bugbot is the option if your team already lives entirely in Cursor. Greptile is worth considering for codebase-graph context, though it scored 23.61% in the same benchmark and bills $30 per seat with 50 credits included and $1 per credit beyond that.
Are Greptile alternatives worth evaluating on benchmark results alone?
No, but benchmark results narrow the field quickly. Greptile detected 23.61% of the bugs in the 2025 benchmark on the 72 it was evaluated against before our access was disabled mid-evaluation. Pair that with the pricing model and the integration surface you need, then run the 20-pull-request test on your own repository, since a public benchmark cannot know your conventions.
Does benchmark performance predict GitHub PR review quality?
Partly. Benchmark recall and precision predict how an AI code reviewer behaves on diffs that resemble the dataset, and MacroscopeBench is deliberately built from real production history to keep that resemblance high. What it cannot capture is your repository's conventions, your team's tolerance for noise, or the custom rules you care about. That is the gap Check Run Agents fills, and the reason the final evaluation should be 20 of your own pull requests.
What does an AI code review cost per pull request?
On MacroscopeBench, model inference ranges from $0.38 to $7.87 per review task depending on the model and effort level, which is the raw cost before any product wraps it. Macroscope's GitHub code review is billed at $0.05 per KB of diff with a 10 KB minimum, which works out to about $0.95 for a typical pull request. CLI reviews use Agent Credits at $0.01 each with 1,000 free per month.
Why does effort level change benchmark results so much?
Effort level controls how much reasoning the model spends per task. Claude Opus 5 moves from 59.4 at low effort to 69.9 at high effort, a 10.5 point swing, which is larger than the gap between several different model families on the same board. That is worth knowing when you read any AI model comparison that does not state the effort setting, because the same model can appear near the top or near the bottom depending on a parameter nobody disclosed.
How many bugs are in the MacroscopeBench dataset?
Over 12,000 validated bugs drawn from more than 1,500 public repositories across 14 languages. Each bug is anchored to the commit that introduced it and to the commit where the project later fixed it, which is what makes the ground truth verifiable rather than assumed. Individual benchmark runs sample from this pool, and control commits with no known defect are included so precision can be scored.
How often is the AI code review benchmark updated?
New models are run through MacroscopeBench as they ship, and the live leaderboard reflects the current set of configurations. It doubles as the regression suite for Macroscope Code Review, so it also runs against prompt and pipeline changes before they reach production rather than only when a new model appears.
Can I use the benchmark to choose a model for my own code review pipeline?
Yes, and the severity filter is the part to use. Decide first whether you are optimizing for catching everything or for being believed, then read the column that matches: recall for coverage, precision and signal-to-noise for trust, cost and duration for what you can actually run on every pull request. Then validate the choice on 20 of your own pull requests, because a public benchmark is an argument about method and your repository is the evidence.