AI Code Review Case Study: How Pydantic Auto-Approves 91% of Pull Requests
Pydantic, the most depended-on library in Python, uses AI code review to auto-approve ~91% of its pull requests and ships 4x more code per week. A named case study with the mechanics: Check Run Agents, Approvability, and agent orchestration.
Pydantic, the type-safety layer underneath most of Python, uses AI code review to auto-approve roughly 91% of its pull requests without a human reviewer. Its engineers went from merging 100-150 pull requests a week to about 500, a 4x increase, without adding reviewers. This is a named, on-the-record AI code review case study with real numbers, not a vendor benchmark.
Most AI code review case studies are anonymous ("a leading fintech"), unquantified, or both. This one is neither. Pydantic is a company you can go look at: 16.9 billion downloads, 900,000+ dependent GitHub repositories, and the library that sits underneath the OpenAI SDK, the Anthropic SDK, Google ADK, LangChain, and LlamaIndex. This guide covers what Pydantic measured, how the AI code review setup actually works, and how to reproduce it on your own GitHub repositories.
TL;DR — Pydantic's AI code review results
- ~91% of pull requests are auto-approved by AI code review. Macroscope's Approvability check gives the +1 on safe changes and escalates the rest to human reviewers.
- 4x more code shipped per week. Pull request throughput went from 100-150/week to roughly 500/week, with the same engineering team.
- Their review standards were learned, not hand-written. Pydantic pointed an agent at a year of human PR comments and turned what its senior and principal engineers flagged most often into Check Run Agents.
- Zero per-repo tuning. The team's hard requirement was a GitHub code review tool they could turn on everywhere, for everyone, without tuning it per repository or per engineer.
- A full epic now ships in an afternoon. With Murmur agent orchestration, work that took 15 hours takes 2.
- Context matters: Intercom has publicly reported ~19% of its pull requests auto-approved with no human in the loop. Pydantic's ~91% is the highest published figure we are aware of.
Why This AI Code Review Case Study Is Worth Reading
The hard part of AI code review is not finding bugs. It is earning enough trust to remove a human from the merge path. Nearly every AI code review tool will post comments on a pull request. Very few teams let one approve a pull request, because an approval is a claim about risk rather than a suggestion, and a wrong approval ships to production.
That is why the auto-approval rate, not the comment count, is the number that separates AI code review tools that changed how a team works from AI code review tools that added a bot to the PR thread. Pydantic's ~91% is a measurement of delegated trust.
It is also worth noting what is not in this case study. Neither CodeRabbit nor Greptile publishes a named customer case study reporting an auto-approval rate. The public comparisons between them are bug-detection bake-offs on open-source pull requests. Bug detection is necessary, but it is the input to the question teams actually care about: how much of my review queue can I stop reading?
Who Pydantic Is, and Why Their Bar Is High
Pydantic is the type-safety layer most of Python runs on. Samuel Colvin released it in 2017, and today nearly every Python agent framework is built on it. As Bill Easton, Head of Product at Pydantic, puts it: "Pydantic allows my spaghetti code to talk to your fettuccine code."
The company now builds three things: Pydantic Validation, the open-source library; Pydantic AI, a framework for building agents in Python; and Logfire, an observability platform for running agents at scale.
This matters for an AI code review case study because of the blast radius. When 900,000 repositories depend on your library, a regression is not an internal incident. A team in that position does not hand merge authority to an AI code review tool casually, which makes their 91% a meaningful data point rather than an enthusiastic early-adopter anecdote.
The Problem: Review Became the Bottleneck
Coding agents broke the economics of human code review before they broke anything else. Once a team adopts agents seriously, pull requests arrive faster than engineers can read them. The constraint moves from writing code to reviewing it, and the review queue becomes the place throughput goes to die.
This is now a well-documented industry pattern. GitHub reports more than one in five code reviews on GitHub involves an agent, and research on agent-authored pull requests found 58.77% were reviewed only by agents, with no human reading them at all.
That second number is the danger. The realistic choice is not "human review" versus "AI code review." It is deliberate AI code review with a policy and an escalation path, versus pull requests quietly merging because nobody had time. Pydantic chose the first.
Why Pydantic Chose Macroscope for AI Code Review
Pydantic evaluated essentially every AI code review tool on the market. In Easton's words: "If you had Googled code review, we tried every single option that showed up in the results."
Two things decided it.
Specificity of feedback. The team wanted AI code review that pointed at the exact line, named the exact problem, and explained the fix, rather than a summary of the diff. Easton described the review output as pointing "at like the specific line, the specific problem, and tell me how to fix it."
No per-repo tuning. This was the hard requirement, and it is the one most GitHub code review tools fail. Pydantic wanted something it could "just turn on on everything and everywhere and for every person" and trust. As Easton put it: "We didn't need to do tuning on a per repo basis or per person basis. And that really allowed us to move really quickly."
That requirement is worth dwelling on, because it is a procurement criterion disguised as a technical one. An AI code reviewer that needs a week of configuration per repository does not scale to an organization. It scales to a pilot.
How Pydantic Got to 91%: Check Run Agents Learned From a Year of Review Comments
Getting to a 91% auto-approval rate took more than bug detection. It required encoding what Pydantic's own reviewers care about.
The team's approach was unusually clever, and it is the most reproducible part of this case study: they used an agent to write their review standards for them.
Pydantic pointed an agent at a year's worth of human-written pull request comments, extracted the things its senior and principal engineers flagged most often, and grouped those into Check Run Agents — customer-defined AI checks that run on every pull request as a GitHub check run.
One of the resulting checks is an AI slop detector. Easton describes it as a catalogue of "all the things that annoy different team members about having AI work in the repo." Pydantic now runs a large suite of these across the codebase, covering style, security, performance, and safety.
Why this works better than writing the rules by hand: a year of review comments is a record of what your team actually enforces, including the conventions nobody ever wrote down. A hand-authored style guide is a record of what someone once intended to enforce. The first is a far better training input for AI code review.
How Approvability Decides Which Pull Requests Are Safe
Approvability is the AI code review check that answers a different question from bug detection: given this pull request's scope, complexity, change type, and runtime impact, is it safe to merge without a human?
Easton describes the effect as sectioning the PR queue into two camps: "Here are the pull requests that we can just merge, that are safe to merge, that pass correctness checks, that pass check runs, and don't impact the system in a way that's dangerous."
Three hurdles clear before Macroscope posts a GitHub approval:
| Hurdle | What it checks | Default |
|---|---|---|
| Ownership | Does the author own the files they changed, per CODEOWNERS? | Off by default |
| Eligibility | An LLM agent classifies the PR against the risk policy | On |
| Correctness | The AI code review pass found no MEDIUM-or-higher bugs | On |
All three must pass. A clean-looking refactor with a real bug in it is still a pull request that needs a human, which is why correctness is a hard gate rather than a signal. Teams override the default policy with a .macroscope/approvability.md file at the repo root, and their rules win. Full mechanics are in What Is Approvability?.
Notably, Pydantic set out to hit 80% — an 80/20 split of AI-reviewed to human-reviewed pull requests. The measured rate came in higher than the target.
How 91% Compares to Published Auto-Approval Rates
Pydantic's ~91% is, as far as we can tell, the highest publicly reported AI code review auto-approval rate. For honest context:
| Organization | Auto-approval rate | What is being measured |
|---|---|---|
| Pydantic (Macroscope) | ~91% | PRs approved by Approvability, rest escalated to humans |
| Intercom | ~19% | PRs auto-approved with no human reviewer in the loop |
| Microsoft (internal) | ~90% assisted | PRs an AI reviewer supports, not PRs it approves alone |
| CodeRabbit | Not published | No named customer auto-approval rate |
| Greptile | Not published | No named customer auto-approval rate |
These are not perfectly like-for-like, and it would be dishonest to present them as such. Intercom's figure and Pydantic's are the closest pair, and they are measuring nearly the same thing on different codebases with different risk policies. Microsoft's ~90% measures AI assistance, which is a much lower bar than approval. The point of the table is not that one number beats another. It is that published, named, quantified auto-approval data is rare, and most AI code review vendors do not have any.
Keeping Engineering and Product in Sync
Pydantic also uses Status to see where engineering effort is going and how it maps to product goals.
Easton found this most valuable during his first weeks and months at the company: understanding where the engineering effort was going, and how it related to the product goals, was the thing that told him what the team was actually building. That picture would otherwise take months of meetings to assemble.
This is the less-discussed half of AI code review. A system that reads every pull request already knows what the organization is working on. Reporting that back is nearly free once the review infrastructure exists.
Shipping a Full Epic in an Afternoon With Agent Orchestration
Once AI code review removed the review bottleneck, the constraint moved back upstream to writing the code. Pydantic became an early customer of Murmur, Macroscope's agent orchestration platform.
The team plans an epic in GitHub, agrees on it, then spawns 15-20 Murmur agents against the parallelizable work — each in its own cloud VM running Pydantic's full development stack. Easton contrasts this with the alternative: "nurturing one cloud session through the entire epic over the course of several days or several weeks."
The loop closes on itself. Murmur reacts to the AI code review comments, the review runs again on the update, and the check runs re-evaluate. By the time a person opens the pull request, as Easton puts it, "it's not a draft":
"It went through all the review. It went through all the check runs. It went through all the CI, and what you're looking at is a completed PR that's ready to merge. That is neat when you have one or two, but it's extremely powerful when you have 20 or 30 tasks running and you're not babysitting 20 or 30 terminals. You're just kind of coming in and seeing that 10 of them are ready."
Work that took the team 15 hours now takes 2. During the interview itself, Easton mentioned having about 25 Murmur tasks running — while drinking a coffee.
What This Means If You Write Python
Pydantic is a Python shop, and Python is where AI code review has the most room to help. Type safety in Python is opt-in, a large share of agent-generated code is Python, and the runtime will not catch for you what a compiler catches in Rust or Go. That puts more weight on the review layer.
The mechanics in this case study are language-agnostic — Macroscope reviews across languages, and builds an AST-level reference graph per language — but the value of catching a cross-file contract break before it merges is highest in a dynamically typed language with 900,000 downstream dependents.
Macroscope vs CodeRabbit vs Greptile for Auto-Approval
For teams evaluating AI code review tools with this use case in mind:
| Macroscope | CodeRabbit | Greptile | |
|---|---|---|---|
| Auto-approve safe PRs | Yes — Approvability, with a policy file | Review comments | Review comments |
| Custom checks from your own standards | Yes — Check Run Agents as GitHub check runs | Rule config | Rule config |
| Per-repo tuning required | No | Varies | Varies |
| Pricing model | Usage-based, no per-seat fee | Per seat | Per seat |
| Agent orchestration | Yes — Murmur | No | No |
| Named customer auto-approval data | Yes — this case study | Not published | Not published |
Deeper comparisons: CodeRabbit alternatives and Greptile alternatives.
The pricing row matters more than it looks when agents enter the picture. Per-seat GitHub code review pricing charges by headcount while your pull request volume is set by agent count. Pydantic 4x'd its PR throughput with the same team — under a per-seat model, that is 4x the review work at the same price to the vendor, which is exactly the pressure that produces quotas and rationing.
How to Reproduce This Setup on Your Repositories
The Pydantic configuration is four steps, and none require per-repository tuning.
- Install AI code review across your repositories. Connect the GitHub app and enable review on everything rather than piloting on one repo. Blanket coverage is what made Pydantic's rollout work. See AI code review on GitHub in 5 minutes.
- Mine your own review history. Point an agent at a year of PR comments and cluster what your senior engineers repeatedly flag. That output is your check-run spec, and it is more accurate than any style guide you would write from memory.
- Turn those clusters into Check Run Agents. Add
.macroscope/*.mdfiles describing each check. They run as GitHub check runs on every pull request. - Enable Approvability last, and conservatively. Start with the default policy, watch what it approves for a few weeks, then widen it with
.macroscope/approvability.mdas trust builds. Approvability is off by default for exactly this reason.
Only after those four is agent orchestration worth adding. Spawning 20 agents against a codebase whose review layer you do not yet trust just moves the bottleneck without removing it.
Frequently Asked Questions
What percentage of pull requests can AI code review safely auto-approve?
Pydantic runs at roughly 91% with Macroscope's Approvability check. Intercom has publicly reported around 19%. The honest answer is that the ceiling depends on your risk policy and how much of your review standard you have encoded, not on the AI code review tool alone. Teams typically start conservative, measure what gets approved, and widen the policy as trust builds.
Is it safe to let AI approve pull requests without a human reviewer?
It is safe when approval is gated rather than assumed. Macroscope requires three hurdles to clear: file ownership, an LLM risk verdict against an explicit policy, and a clean correctness pass with no MEDIUM-or-higher bugs. Anything that fails any hurdle goes to a human. The alternative most teams actually face is worse — research found 58.77% of agent-authored pull requests were reviewed by no human at all, which is auto-approval without a policy.
What is the best AI code reviewer for Python teams?
Python benefits disproportionately from AI code review because type safety is opt-in and the runtime catches less than a compiler would. Pydantic, which is the type-safety layer for most of Python, evaluated the field and chose Macroscope, citing line-specific fixes and the fact that it required no per-repository tuning. See AI code review vs human code review for the tradeoffs.
How did Pydantic build its custom AI code review checks?
They had an agent read a year of human-written pull request comments, identify what their senior and principal engineers flagged most often, and group those into Check Run Agents. One resulting check catalogues the things that annoy team members about AI-written code in the repo. This is more accurate than hand-writing rules, because review history records what a team actually enforces rather than what it intended to.
What are the best CodeRabbit alternatives for teams that want auto-approval?
CodeRabbit posts review comments but does not publish a named customer auto-approval rate. If the goal is removing pull requests from the human review queue rather than adding commentary to them, look for an explicit approvability primitive with a configurable policy file. Macroscope is the option built around that, and it prices usage-based rather than per seat. Full list: best CodeRabbit alternatives.
How is this different from Greptile?
Greptile builds a codebase-aware graph and is strong at bug detection in a single pass, with a correspondingly higher false-positive rate in public comparisons. The difference relevant to this case study is the decision layer: bug detection tells you what is wrong, approvability tells you whether the pull request is safe to merge anyway. Pydantic needed the second. See Greptile alternatives.
Does AI code review require tuning for each repository?
It should not, and this was Pydantic's stated hard requirement. They wanted something they could turn on everywhere, for everyone, with no per-repo or per-person tuning. Tools that need significant per-repository configuration tend to stall at the pilot stage, because the configuration cost scales with the number of repositories.
How much faster does a team ship with AI code review?
Pydantic went from 100-150 pull requests a week to roughly 500, a 4x increase, with the same engineers. Separately, adding agent orchestration took epic work from 15 hours to 2. The review gain and the authoring gain compound: removing the review bottleneck is what makes increased authoring throughput usable.
What happens to pull requests the AI does not approve?
They escalate to human reviewers, which is the point. The value is not that humans stop reviewing — it is that the ~9% they do review is the slice where judgment is actually required, instead of a queue where the risky changes are buried among doc fixes and test additions.
Can AI code review fix the bugs it finds, not just report them?
Yes. Macroscope's Fix It For Me applies fixes for review comments and CI failures directly to the branch. In Pydantic's setup the loop is tighter still: Murmur agents react to review comments, push updates, and the AI code review runs again on the new commit until the pull request is clear to merge.
How does AI code review handle a 4x increase in pull requests without a price spike?
That depends entirely on the pricing model, and it is the question to ask any vendor before agents multiply your PR count. Per-seat pricing charges by headcount while agents set your volume, which is what produces quotas. Macroscope prices on usage, so cost tracks the reviews actually performed.
Where can I watch the full Pydantic interview?
The complete conversation with Bill Easton, Head of Product at Pydantic, is on YouTube. The written case study is at How Pydantic Merges 91% of Their Pull Requests Without a Human.
