MacroscopeBench: Benchmarking Code Review
A deep dive on MacroscopeBench, the benchmark we built from real bugs in open source repositories to measure whether a code reviewer would have caught them at the moment they were introduced.
MacroscopeBench is a proprietary benchmark that we built to allow us to assess real-world code review performance across different models and code review systems. The primary goal of MacroscopeBench is to evaluate whether a set of real bugs, which were impactful enough to require a fix from open source maintainers or contributors, could have been detected at the time that they were introduced. We use MacroscopeBench daily to build our Code Review product, optimize our Detection Modes and evaluate new models.
With MacroscopeBench now the only trusted benchmark for code review on the Fireworks' Specialized Intelligence Index, we wanted to do a deep dive on our benchmark and share more details on how we built it and how it works.
| Model | ||||||
|---|---|---|---|---|---|---|
GPT-6 Astra max effort | 78.0 | 67.8% | 91.7% | 6.26 | $7.87 | 3m 49s |
GPT-5.6 Sol high effort | 77.8 | 70.8% | 86.2% | 3.73 | $3.90 | 3m |
GLM 5.3 max effort | 77.4 | 76.4% | 78.4% | 2.24 | $3.86 | 12m 30s |
Grok 4.6 xhigh effort | 75.9 | 69.7% | 83.3% | 3.67 | $4.68 | 14m 41s |
Grok 4.6 high effort | 75.1 | 69.9% | 81.1% | 3.17 | $3.51 | 15m 30s |
Kimi K3 max effort | 72.9 | 66.7% | 80.3% | 2.18 | $2.82 | 8m 13s |
DeepSeek V4.1 Flash max effort | 72.0 | 83.1% | 63.5% | 1.27 | $0.75 | 16m 17s |
DeepSeek V4.1 Flash low effort | 70.7 | 77.5% | 65.0% | 1.37 | $0.38 | 8m 9s |
DeepSeek V4.1 Flash high effort | 70.5 | 77.5% | 64.6% | 1.33 | $0.38 | 7m 40s |
Claude Opus 5 high effort | 69.9 | 76.6% | 64.2% | 1.20 | $6.50 | 4m 18s |
Claude Opus 5 medium effort | 65.5 | 68.8% | 62.5% | 1.18 | $3.50 | 2m 12s |
Claude Opus 5 low effort | 59.4 | 57.4% | 61.5% | 1.12 | $1.40 | 56s |
Real bugs, not synthetic data
From the beginning, we made the strategic decision that the dataset of our benchmark should be composed of real bugs created by real people, rather than synthetic bugs produced by an LLM. The reason is simple: our benchmark measurements needed to approximate the performance of the Macroscope code review product as closely as possible. So we built our dataset by searching for pull requests that fixed real issues in popular open source repositories and tracing those issues back to the commit that introduced them. This allows us to evaluate code review performance based on whether the reviewing LLM is able to correctly identify a real issue that was important and impactful enough to specifically fix at the time that the issue was introduced. Over time, we have found this to be a much better way to assess the models' performance on doing the reviews it will be asked to in production. To further harden the results, we also included 'control' commits which have no known bugs in our dataset. Bugs reported by reviewers on these commits are assessed using the same validity criteria as bug reports on other commits.
Diverse languages and severity
Our dataset is gathered from a broad range of popular open source repositories spanning multiple projects, domains and languages. The full benchmark dataset contains over 12,000 validated bugs, drawn from over 1,500 public repositories across 14 languages. The benchmark subset used for the Specialized Intelligence Index comprises 195 commits: 144 that introduced at least one known bug subsequently fixed by human maintainers or contributors, and 51 control commits.
We also carefully analyzed the harm that could be caused by each of the known bugs in our dataset, labelling each with one of the following categories:
- Critical: Permanent loss or compromise, such as unrecoverable data loss or unauthorized access.
- High: Serious disruption, such as crashes, hangs, or weakened data integrity.
- Medium: Degraded functionality or incorrect behavior in core paths without lasting damage.
- Low: Minor functional impact or defects affecting rarely reached edge cases.
The severity of a bug in the dataset is determined and fixed when it enters the dataset. It does not change depending on which model is being evaluated, so we can compare performance of the model across each severity bucket. We intentionally determined the distribution of bug severity in our dataset to ensure that the benchmark is useful for distinguishing reviewer performance, even if this means that the exact distribution of bugs may not perfectly reflect the distribution observed across the repositories that we looked at.
Evaluating review performance
For each task, we check out the repository at the evaluated commit and use the code reviewer being tested to review the diff between that commit and its parent. The reviewer is never told whether a bug exists, how many there are, or where to look. Each task is evaluated independently. The reviewer receives repository and diff context but is not supplied with the benchmark's known-bug descriptions or the subsequent pull request that introduces the fix.
Each task is then evaluated using two LLM judges, each of which has access to tools that allow it to explore the codebase and search official documentation.
Recall: The first LLM judge matches reported findings against descriptions of known bugs in the dataset. These descriptions are derived from the corresponding fixing pull requests. Each known bug receives detection credit at most once per run, even if multiple reports describe the same defect. This means that recall cannot exceed 100%.
Precision: A second, independent LLM judge then evaluates all of the bugs reported by the reviewer during the task, which covers all output from the reviewer. This judge only considers a bug to be valid if it is an objective correctness issue that was introduced (or exposed) by the commit that is being reviewed. All reviewers are prompted to report only valid objective correctness bugs. Therefore, if the reviewer reports a bug that relates to code style, optimizations or other potential subjective improvements or changes, it is considered to be invalid by this judge. All benchmark dataset bugs are valid objective correctness issues that have been independently verified by an LLM judge during benchmark dataset compilation.
Score: For each of three independent runs over the full task set, we calculate the harmonic mean of known-bug recall and overall bug detection precision. The reported score is the arithmetic mean of these three scores. We believe this is the best overall measure of code review performance because it rewards detecting known bugs while penalising noise from invalid, subjective, and out-of-scope findings.
Comparing cost and latency
In addition to evaluating the performance of a reviewer against our benchmark, we also assess each reviewer in terms of cost and latency, as these are also critical metrics to consider when developing our Code Review Detection Modes or evaluating a new model.
Cost: Cost is the average model inference cost per code review task, calculated using published API prices and accounting for intra-task caching. All judging and other benchmark setup and infrastructure costs are excluded.
Latency: Latency is measured in terms of median wall-clock seconds elapsed per code review task, excluding any time taken for judging or other benchmark evaluation or setup steps.
Ensuring fair comparison
Language models are not deterministic, and code review makes that variation visible. The same model, on the same diff, with the same prompt, can find a defect on one run and miss it on the next. We have measured variation of several percentage points across repeated runs on a fixed task set. That is large enough to make a single-run comparison unreliable.
We therefore evaluate each reviewer across three independent runs, using an arithmetic mean for most of our metrics. For the model comparisons published in Fireworks' Specialized Intelligence Index, we use the same prompt, tool schema, harness and judge configuration across all models and runs.
Limitations
- We specifically aim to evaluate models under common conditions by using the same prompt and harness for all models; however, this does mean that the benchmark does not capture any gains that may be available from model-specific prompt optimization or harness tuning.
- Recall, precision, and severity are assessed using LLM judges. The known bugs in the benchmark dataset are real defects subsequently fixed by human maintainers or contributors, but detection matches, reported-bug validity, and severity labels are determined by LLM judges without human adjudication.
- The severity composition of the dataset may not reflect real-world distribution. The dataset is designed to allow for discrimination between models, not to reflect real-world bug distributions.
- Subjective issues or issues that do not strictly relate to correctness are penalised. Any bug reports that relate to style, documentation quality, optimization opportunities or other non-objective or non-correctness issues are considered invalid and count against precision.
Officially introducing MacroscopeBench
We are excited to share more about MacroscopeBench and make it available for people to reference as part of the Fireworks Specialized Intelligence Index. We hope this can help others select the best model for their task, the same way it has for us over the past year. We will continue to improve MacroscopeBench over time, in addition to continuing to leverage it to improve the performance of Macroscope Code Review. If you are someone who would like to get access to and leverage our dataset, or have any questions regarding the details above, please reach out to our team, we'd love to hear from you.
