MacroscopeBench
MacroscopeBench tests whether code reviewers catch real bugs at the commits that introduced them. It’s the benchmark we use to build and tune Macroscope Code Review.
Read about the benchmarkModel performance
Higher score means stronger combined precision and recall.
vs
| Model | ||||||
|---|---|---|---|---|---|---|
Claude Opus 5.5 max effort | 83.9 | 83.8% | 84.0% | 3.01 | $11.58 | 10m 8s |
GPT-6 Sol max effort | 80.3 | 73.8% | 88.0% | 3.94 | $3.68 | 4m 57s |
Claude Opus 5.5 xhigh effort | 78.9 | 75.0% | 83.2% | 2.95 | $3.51 | 2m 52s |
GPT-6.1 Sol max effort | 78.5 | 69.2% | 90.7% | 6.42 | $1.05 | 3m 59s |
GPT-6 Astra max effort | 78.0 | 67.8% | 91.7% | 6.26 | $7.87 | 3m 49s |
GPT-5.6 Sol high effort | 77.8 | 70.8% | 86.2% | 3.73 | $3.90 | 3m |
GLM 5.3 max effort | 77.4 | 76.4% | 78.4% | 2.24 | $3.86 | 12m 30s |
GPT-6.1 Sol xhigh effort | 77.2 | 66.9% | 91.4% | 6.74 | $0.79 | 2m 43s |
GPT-6 Sol xhigh effort | 77.2 | 69.0% | 87.6% | 3.86 | $2.00 | 3m 6s |
GPT-6.1 Sol high effort | 76.7 | 66.2% | 91.3% | 6.93 | $0.56 | 1m 52s |
Grok 4.6 xhigh effort | 75.9 | 69.7% | 83.3% | 3.67 | $4.68 | 14m 41s |
Grok 4.6 high effort | 75.1 | 69.9% | 81.1% | 3.17 | $3.51 | 15m 30s |
GPT-6 Sol high effort | 74.3 | 65.5% | 85.9% | 3.62 | $1.23 | 2m 3s |
GPT-6.1 Sol medium effort | 73.5 | 61.6% | 91.3% | 7.47 | $0.28 | 1m 6s |
Claude Opus 5.5 high effort | 73.1 | 68.1% | 78.9% | 2.35 | $1.30 | 1m 3s |
Kimi K3 max effort | 72.9 | 66.7% | 80.3% | 2.18 | $2.82 | 8m 13s |
Grok 4.6 medium effort | 72.8 | 67.1% | 79.5% | 3.11 | $2.56 | 8m 39s |
GPT-6 Luna max effort | 72.5 | 61.1% | 89.1% | 4.28 | $0.13 | 4m 1s |
DeepSeek V4.1 Flash max effort | 72.0 | 83.1% | 63.5% | 1.27 | $0.75 | 16m 17s |
Gemini 3.8 Flash medium effort | 70.7 | 64.1% | 78.9% | 2.34 | $3.38 | 11m 3s |
DeepSeek V4.1 Flash low effort | 70.7 | 77.5% | 65.0% | 1.37 | $0.38 | 8m 9s |
GPT-6.1 Sol low effort | 70.6 | 58.1% | 89.9% | 6.63 | $0.16 | 46s |
DeepSeek V4.1 Flash high effort | 70.5 | 77.5% | 64.6% | 1.33 | $0.38 | 7m 40s |
Claude Opus 5.5 medium effort | 70.1 | 63.4% | 78.3% | 2.35 | $0.97 | 44s |
Claude Opus 5 high effort | 69.9 | 76.6% | 64.2% | 1.20 | $6.50 | 4m 18s |
GLM 5.3 high effort | 69.4 | 64.6% | 75.0% | 1.94 | $2.02 | 5m 29s |
GPT-6 Sol medium effort | 69.2 | 59.7% | 82.2% | 3.08 | $0.69 | 1m 31s |
Kimi K3 high effort | 68.6 | 62.0% | 76.6% | 1.89 | $1.90 | 4m 21s |
GPT-6 Luna xhigh effort | 68.2 | 56.3% | 86.6% | 3.61 | $0.06 | 2m 2s |
GPT-5.6 Sol low effort | 66.1 | 55.1% | 82.6% | 3.17 | $0.81 | 1m 4s |
GPT-6 Luna high effort | 65.6 | 53.0% | 86.0% | 3.50 | $0.05 | 1m 40s |
Claude Opus 5 medium effort | 65.5 | 68.8% | 62.5% | 1.18 | $3.50 | 2m 12s |
GPT-6 Luna medium effort | 62.4 | 50.9% | 80.5% | 2.60 | $0.03 | 1m 14s |
GPT-6 Sol low effort | 61.8 | 52.1% | 75.9% | 2.35 | $0.36 | 58s |
Claude Opus 5.5 low effort | 61.4 | 52.3% | 74.3% | 2.21 | $0.68 | 29s |
Claude Opus 5 low effort | 59.4 | 57.4% | 61.5% | 1.12 | $1.40 | 56s |
Grok 4.6 low effort | 57.1 | 52.5% | 62.5% | 1.30 | $0.30 | 55s |
GLM 5.3 low effort | 54.6 | 50.0% | 60.0% | 1.17 | $0.41 | 1m 22s |
Kimi K3 low effort | 53.5 | 45.1% | 65.7% | 1.37 | $0.48 | 1m 3s |
Gemini 3.8 Flash low effort | 51.3 | 44.4% | 60.8% | 1.11 | $0.90 | 2m 38s |
GPT-6 Luna low effort | 44.7 | 38.0% | 54.5% | 1.00 | $0.02 | 42s |
Metric Definitions
Every model is run 3 independent times through the benchmark to account for nondeterminism and diverging results. Every metric, unless stated otherwise, is the arithmetic mean of the three independent runs.
- Score: The harmonic mean of known-bug recall and overall bug detection precision.
- Recall: The detection rate of the known bugs within the benchmark dataset. A bug is credited when an LLM judge determines that it matches a bug that was later explicitly identified and fixed in the codebase. Credit is per bug, so several bug reports describing the same defect count once and recall cannot exceed 100%.
- Precision: The percentage of all bugs reported by the model across the full benchmark that an LLM judge determines to be valid bugs that are introduced (or exposed) by the specific commit being reviewed in each task. The judge can explore the codebase when assessing validity. Invalid, subjective, and out-of-scope findings count against precision.
- Severity buckets: All catalogued bugs and raised issues by the reviewer are assigned a severity grade based on potential harm of the bug. Selecting a severity bucket will update Score, Recall, and Precision to only consider bugs with that severity ranking.
- Critical: Permanent loss or compromise, such as unrecoverable data loss or unauthorized access.
- High: Serious disruption, such as crashes, hangs, or weakened data integrity.
- Medium: Degraded functionality or incorrect behavior in core paths without lasting damage.
- Low: Minor functional impact or defects affecting rarely reached edge cases.
- Signal / Noise: The ratio of useful comments generated by the reviewer (signal) vs unuseful comments (noise). Useful comments are all valid correctness issues detected with a severity grade of Medium or higher. Unuseful comments are any issues deemed to be invalid for the task, or of Low severity.
- Cost: The model inference cost in US dollars per code review task. Judging costs are excluded.
- Duration: The median wall-clock seconds elapsed per code review task, excluding any time taken for judging or other benchmark evaluation or setup steps.