AI code review benchmark
Which models actually catch real bugs, measured on 30 merged pull requests from production open-source projects against 95 confirmed bugs. Every run is versioned in the repo, so you can check the numbers or re-score them yourself.
- harness
- kodus
- judge
- claude-haiku-4-5
- execution
- replay
- pull requests
- 30
- confirmed bugs
- 95
- bugs found
- 42 of 95
- recall
- 44.2%
- precision
- 43.6%
How the benchmark works
Every model reviews the same 30 real PRs, against the same human-authored golden comments, judged the same way. One run per model, at vendor defaults, so the numbers are what you get out of the box.
Real PRs, real bugs
30 merged pull requests from 5 production OSS repos, each with human-authored review comments as ground truth. 95 golden bugs in total.
Deterministic replay
Each model runs the same production review agent, with tool calls replayed against a frozen snapshot of the repo. No live network, and no non-determinism from the codebase changing under it.
One judge, every finding
claude-haiku-4-5 decides whether each reported finding describes the same underlying issue as a golden comment. Micro-averaged: true and false positives are summed across all PRs before computing precision and recall.
Publish the artifacts
Every submission and scorecard is versioned in the repo. Re-scoring never requires re-running a model, only the judge call is repeated.
What the numbers mean
Recall. How many of the known bugs the model actually found.
Precision. Of what it reported, how much was real. A model that talks more finds more but also misfires more.
F1. Harmonic mean of both, equal weight. What the leaderboard ranks by, so talking more is never free.
95% CI. How much recall would move on a different sample of 30 PRs. Most of these intervals overlap, so the exact rank is noise, not signal.
What this doesn't measure
Model run-to-run variance. One pass per model, so the review agent itself only runs once per entry. Judge noise (re-scoring the same submission) is measured separately where available. Treat close scores as tied either way.
Harnesses other than Kodus's, so far. Every entry to date runs on the same harness, so what you are comparing is models inside one review agent. The submission contract is open, and other harnesses get measured the same way once they are submitted.
Source repositories
Merged PRs across 5 languages (go, java, python, ruby, typescript). Same set, same golden comments, for every model, so the comparison is fair even when the result isn't flattering.
Global leaderboard
What the results say so far
No model finds even half. The best recall in this benchmark is under 45%. Code review has a lot of headroom left.
Precision and recall trade off. The model that talks the least is often the most precise. Different products, not different quality.
Bring your own harness. Submissions are a documented JSON contract. Submit a PR against the same cases.
This benchmark is run and published byKodus, an AI code review company, on Kodus's own harness. Every entry so far runs on that harness, so what the ranking compares is models inside it. See theleaderboardfor the full disclosure.