AI Code Review False Positives and Signal-to-Noise: How Many Comments Are Useful?
A code review tool that produces 10 comments per PR and catches 45% of bugs is harder to use than one that produces 2.5 comments and catches 48%. Signal-to-noise matters as much as detection rate.
A code review tool that catches 45% of bugs but produces 10 comments per PR creates more work than one that catches 48% with 2.5 comments per PR. The first tool forces reviewers to wade through noise. The second tells them what matters.
Signal-to-noise is often the deciding factor in whether a team adopts AI code review or abandons it. High comment volume trains developers to ignore the tool. Low volume with high accuracy trains them to trust it.
How does comment volume affect AI code review usefulness?
More comments are not better if most of them are not actionable. Every comment demands attention. A developer reading a PR with 10 AI comments has to evaluate each one, decide whether it is valid, and either fix it or dismiss it. That evaluation time compounds across every PR.
On Macroscope's 118-bug benchmark, comment volume varied dramatically:
| Tool | Comments per PR | Detection rate | Comments per bug detected |
|---|---|---|---|
| Macroscope | 2.55 | 48.31% | 5.3 |
| CodeRabbit | 10.84 | 45.76% | 23.7 |
| Greptile | 3.08 | 23.61% | 13.1 |
| Cursor Bugbot | 0.91 | 42.37% | 2.1 |
| Graphite Diamond | 0.62 | 18.26% | 3.4 |
Macroscope produced 2.55 comments per PR while catching 48.31% of bugs. CodeRabbit produced 10.84 comments per PR while catching 45.76%. That is a 4x difference in comment volume for a lower detection rate.
What is precision in AI code review?
Precision measures what fraction of a tool's comments matched changes the developer actually made. High precision means fewer false alarms. A tool with 80% precision is right 4 out of 5 times. A tool with 60% precision is wrong 2 out of 5 times.
On the Martian Code Review Bench (an independent benchmark), precision scores as of October 2026 were:
| Tool | Precision |
|---|---|
| Greptile | 79.0% |
| Macroscope | 78.5% |
| Cubic | 71.0% |
| CodeAnt AI | 70.5% |
| CodeRabbit | 70.2% |
| GitHub Copilot | 67.4% |
Both Greptile and Macroscope exceeded 78% precision, meaning fewer than 1 in 4 comments were false positives. The Martian benchmark measures precision as the share of review comments that matched a change the developer actually made.
Source: Martian Code Review Bench
What is the difference between precision and detection rate?
Detection rate (recall) measures how many bugs the tool found. Precision measures how many of its comments were correct.
A tool can have high detection and low precision by commenting on everything, catching bugs in the noise but wasting reviewer time. A tool can have high precision and low detection by staying quiet unless very confident, missing bugs but never wasting time.
The ideal is high precision and high detection. The Macroscope benchmark data shows this tradeoff:
- Macroscope: 48.31% detection, 2.55 comments/PR. High detection, high signal-to-noise.
- CodeRabbit: 45.76% detection, 10.84 comments/PR. Slightly lower detection, much higher volume.
- Cursor Bugbot: 42.37% detection, 0.91 comments/PR. Lower detection, very high signal-to-noise.
- Graphite Diamond: 18.26% detection, 0.62 comments/PR. Low detection, low volume.
Cursor Bugbot achieves excellent signal-to-noise (under 1 comment per PR) but catches fewer bugs. Macroscope balances both.
How do I calculate signal-to-noise for my team?
Track how often your team acts on AI comments versus dismissing them. Over a week or month, count:
- Total AI review comments across all PRs
- Comments that led to a code change
- Comments that were dismissed or ignored
The ratio of (2) to (1) is your effective precision. If 30% of comments lead to changes, the tool is producing 70% noise.
You can also track time spent. If developers spend 5 minutes per AI comment evaluating it, and 70% are dismissed, that is 3.5 minutes wasted per comment times comment volume times PRs per week.
What causes false positives in AI code review?
False positives come from context the AI does not have or patterns it over-matches.
Common sources:
- Missing business context. The AI sees code that looks risky but is intentional for this specific use case.
- Over-applying patterns. A pattern that is usually wrong might be correct in this codebase.
- Incomplete codebase understanding. The AI does not see how this code interacts with other parts of the system.
- Style versus substance. The AI flags something as a potential bug that is actually a style preference.
Reducing false positives requires either more context (larger analysis scope, better codebase understanding) or more restraint (commenting only on high-confidence findings).
How does Macroscope reduce false positives?
Macroscope is designed precision-first: every comment is grounded in evidence from the codebase, not from a guess.
Several design choices contribute:
- Full repository context. Macroscope reads beyond the diff to understand how changes affect the broader codebase.
- Evidence linking. Each comment links to the code that supports it, so reviewers can verify.
- Confidence thresholds. Macroscope does not comment on every possible issue, only the ones it has evidence for.
- Feedback loops. When developers dismiss or react to comments, that signal improves future reviews.
The result is 2.55 comments per PR with 48.31% detection, the best balance of volume and accuracy on the benchmark.
How do Check Run Agents affect signal-to-noise?
Check Run Agents let teams define exactly what to check, reducing noise from irrelevant rules.
A general-purpose reviewer might flag things your team does not care about. A Check Run Agent that checks only your specific conventions produces only relevant comments.
For example, a team that always requires feature flags on new endpoints can write a Check Run Agent for that rule. The agent only fires when the rule applies, producing zero comments on PRs that do not add endpoints.
Frequently Asked Questions
What is a good signal-to-noise ratio for AI code review?
A good target is that most comments should lead to a code change or a meaningful discussion. If more than half of AI comments are dismissed, the tool is producing too much noise. Macroscope's 2.55 comments per PR with 48.31% detection suggests most comments are actionable.
How many comments per PR is too many?
It depends on PR size, but double-digit comments per average PR is a red flag. CodeRabbit's 10.84 comments per PR on the benchmark means developers spend significant time evaluating comments. Macroscope's 2.55 is easier to process.
What is precision in AI code review?
Precision is the share of the tool's comments that matched a change the developer actually made. High precision means fewer false positives. On the Martian benchmark, Macroscope achieved 78.5% precision.
What is the difference between precision and recall?
Precision measures how many comments were correct. Recall (detection rate) measures how many bugs were found. A tool can be high on one and low on the other. The ideal is high on both.
Why do some AI code review tools produce so many comments?
Some tools are tuned for recall over precision, flagging anything that might be an issue. This catches more bugs but creates noise. Other tools are tuned for precision, commenting only when confident. The choice is a design decision, not a technical limitation.
How can I reduce false positives from AI code review?
Configure the tool to match your codebase patterns, use Check Run Agents for team-specific rules, and dismiss false positives so the tool can learn. Most tools improve with feedback.
Does comment volume correlate with detection rate?
Not necessarily. CodeRabbit produced 4x the comments of Macroscope but caught fewer bugs. Cursor Bugbot produced very few comments with solid detection. Volume and accuracy are independent dimensions.
How do I evaluate AI code review signal-to-noise before buying?
Run the tool on real PRs and track how many comments lead to changes. Most tools offer trials. A week of real usage tells you more than any benchmark.
Is precision more important than recall?
For day-to-day use, precision often matters more. A tool with high precision and moderate recall gets used because developers trust it. A tool with high recall and low precision gets ignored because it cries wolf.
How does Macroscope compare to CodeRabbit on signal-to-noise?
Macroscope produces 2.55 comments per PR with 48.31% detection. CodeRabbit produces 10.84 comments per PR with 45.76% detection. Macroscope has better signal-to-noise: higher detection with fewer comments.