# CodeReviewBench > An open benchmark measuring how well AI models find real bugs in code review. > Models review merged pull requests from production open-source repositories, > inside the Kodus review harness, in two review modes, and are scored against a > labelled set of the real bugs in those PRs. Maintained by Kodus. Every submission and scorecard is versioned in the public repo. ## What is measured 13 models, 30 merged pull requests, 121 confirmed bugs. Repositories: cal.com, discourse, grafana, keycloak, sentry. Languages: go, java, python, ruby, typescript. Review modes: balanced (the normal review, with one generalist agent.) and heavy (the generalist plus 9 specialist lens agents in parallel, then the same verify and dedup steps.) Judge: Claude Haiku 5.5, validated against Claude Opus 5.5 (agreed on 249 of 252 comments). Runs: 2026-11-07 to 2026-11-09. Data generated 2026-11-09T23:02:28.320Z. Gold set: the same 30 PRs as the earlier version of this benchmark, with the real problems re-labelled by Kodus (Claude Opus plus a second reviewer, debated where they disagreed): 121 valid true positives. Recall is valid true positives found divided by 121. Precision is published comments that match a valid true positive divided by all published comments; each true positive counts once, and a second comment on the same one is a false positive. F1 is the harmonic mean. F2 is the F-score with beta = 2 (popularized for code review by Martian's benchmark): F2 = 5·P·R / (4·P + R). It weighs recall twice as much as precision, because a missed bug costs more than a comment you dismiss. Results are ranked by F2 within each mode. Cost uses each vendor's list price (read 2026-11-07) with prompt cache: (input - cacheRead - cacheWrite) x input price + cacheRead x cache-read price + cacheWrite x cache-write price + output x output price. - Served by AkashML in fp8 (8-bit quantized weights). - Promotional price, valid until 2026-12-31. - Ran at the vendor's default reasoning effort, which reasons very little on this task (about 25 reasoning tokens per PR). ## Current results ### Balanced mode The normal review, with one generalist agent. 1. [Kimi K3.1 Pro](https://codereviewbench.com/model/kimi-k3.1-pro): F2 63.5, recall 68.6%, precision 49.1%, F1 57.2; found 83/121 confirmed bugs with 169 comments; $1.244/PR, $0.450/bug found. 2. [GPT-6.1 Sol](https://codereviewbench.com/model/gpt-6.1-sol): F2 45.9, recall 43.0%, precision 62.6%, F1 51.0; found 52/121 confirmed bugs with 83 comments; $0.787/PR, $0.454/bug found. 3. [Claude Opus 5.5](https://codereviewbench.com/model/claude-opus-5.5): F2 43.9, recall 42.1%, precision 52.6%, F1 46.8; found 51/121 confirmed bugs with 97 comments; $0.715/PR, $0.421/bug found. 4. [GLM-5.3](https://codereviewbench.com/model/glm-5.3): F2 41.7, recall 41.3%, precision 43.5%, F1 42.4; found 50/121 confirmed bugs with 115 comments; $1.171/PR, $0.702/bug found. 5. [DeepSeek V4.1 Flash](https://codereviewbench.com/model/deepseek-v4.1-flash): F2 41.0, recall 40.5%, precision 43.4%, F1 41.9; found 49/121 confirmed bugs with 113 comments; $0.134/PR, $0.082/bug found. 6. [Claude Sonnet 5.5](https://codereviewbench.com/model/claude-sonnet-5.5): F2 40.0, recall 37.2%, precision 57.0%, F1 45.0; found 45/121 confirmed bugs with 79 comments; $0.542/PR, $0.361/bug found. 7. [Muse Spark 1.2](https://codereviewbench.com/model/muse-spark-1.2): F2 36.4, recall 37.2%, precision 33.6%, F1 35.3; found 45/121 confirmed bugs with 134 comments; $0.725/PR, $0.483/bug found. 8. [GLM-5.3 Flash](https://codereviewbench.com/model/glm-5.3-flash): F2 36.3, recall 35.5%, precision 39.8%, F1 37.5; found 43/121 confirmed bugs with 108 comments; $0.098/PR, $0.069/bug found. 9. [DeepSeek V4 Pro](https://codereviewbench.com/model/deepseek-v4-pro): F2 34.5, recall 32.2%, precision 47.6%, F1 38.4; found 39/121 confirmed bugs with 82 comments; $0.490/PR, $0.377/bug found. 10. [Qwen3.8 27B](https://codereviewbench.com/model/qwen3.8-27b): F2 30.9, recall 28.1%, precision 51.5%, F1 36.4; found 34/121 confirmed bugs with 66 comments; $0.198/PR, $0.175/bug found. 11. [MiniMax M3](https://codereviewbench.com/model/minimax-m3): F2 21.9, recall 19.8%, precision 38.1%, F1 26.1; found 24/121 confirmed bugs with 63 comments; $0.154/PR, $0.193/bug found. 12. [Gemini 3.8 Flash](https://codereviewbench.com/model/gemini-3.8-flash): F2 18.2, recall 15.7%, precision 50.0%, F1 23.9; found 19/121 confirmed bugs with 38 comments; $0.348/PR, $0.550/bug found. 13. [GPT-6 Luna](https://codereviewbench.com/model/gpt-6-luna): F2 8.1, recall 6.6%, precision 61.5%, F1 11.9; found 8/121 confirmed bugs with 13 comments; $0.023/PR, $0.086/bug found. ### Heavy mode The generalist plus 9 specialist lens agents in parallel, then the same verify and dedup steps. 1. [Kimi K3.1 Pro](https://codereviewbench.com/model/kimi-k3.1-pro): F2 58.4, recall 67.8%, precision 37.6%, F1 48.4; found 82/121 confirmed bugs with 218 comments; $2.306/PR, $0.844/bug found. 2. [GPT-6.1 Sol](https://codereviewbench.com/model/gpt-6.1-sol): F2 55.4, recall 55.4%, precision 55.4%, F1 55.4; found 67/121 confirmed bugs with 121 comments; $1.741/PR, $0.780/bug found. 3. [GLM-5.3](https://codereviewbench.com/model/glm-5.3): F2 50.2, recall 55.4%, precision 36.6%, F1 44.1; found 67/121 confirmed bugs with 183 comments; $2.539/PR, $1.137/bug found. 4. [Claude Opus 5.5](https://codereviewbench.com/model/claude-opus-5.5): F2 49.4, recall 49.6%, precision 48.8%, F1 49.2; found 60/121 confirmed bugs with 123 comments; $3.100/PR, $1.550/bug found. 5. [DeepSeek V4.1 Flash](https://codereviewbench.com/model/deepseek-v4.1-flash): F2 48.9, recall 52.9%, precision 37.6%, F1 44.0; found 64/121 confirmed bugs with 170 comments; $0.341/PR, $0.160/bug found. 6. [Claude Sonnet 5.5](https://codereviewbench.com/model/claude-sonnet-5.5): F2 48.8, recall 47.1%, precision 57.0%, F1 51.6; found 57/121 confirmed bugs with 100 comments; $1.754/PR, $0.923/bug found. 7. [DeepSeek V4 Pro](https://codereviewbench.com/model/deepseek-v4-pro): F2 46.4, recall 47.9%, precision 41.1%, F1 44.3; found 58/121 confirmed bugs with 141 comments; $1.296/PR, $0.670/bug found. 8. [GLM-5.3 Flash](https://codereviewbench.com/model/glm-5.3-flash): F2 46.3, recall 47.9%, precision 40.6%, F1 43.9; found 58/121 confirmed bugs with 143 comments; $0.221/PR, $0.114/bug found. 9. [Qwen3.8 27B](https://codereviewbench.com/model/qwen3.8-27b): F2 42.3, recall 43.0%, precision 39.7%, F1 41.3; found 52/121 confirmed bugs with 131 comments; $0.409/PR, $0.236/bug found. 10. [Muse Spark 1.2](https://codereviewbench.com/model/muse-spark-1.2): F2 40.6, recall 46.3%, precision 27.3%, F1 34.4; found 56/121 confirmed bugs with 205 comments; $2.072/PR, $1.110/bug found. 11. [MiniMax M3](https://codereviewbench.com/model/minimax-m3): F2 29.0, recall 28.1%, precision 33.3%, F1 30.5; found 34/121 confirmed bugs with 102 comments; $0.318/PR, $0.281/bug found. 12. [Gemini 3.8 Flash](https://codereviewbench.com/model/gemini-3.8-flash): F2 24.2, recall 21.5%, precision 48.1%, F1 29.7; found 26/121 confirmed bugs with 54 comments; $0.738/PR, $0.851/bug found. 13. [GPT-6 Luna](https://codereviewbench.com/model/gpt-6-luna): F2 15.1, recall 13.2%, precision 35.6%, F1 19.3; found 16/121 confirmed bugs with 45 comments; $0.057/PR, $0.107/bug found. ## Interpretation limits One run per model and mode. With 30 PRs, small gaps between models are within sampling noise; each model page shows a 95% bootstrap interval on recall. Kodus runs the benchmark, built the harness and labelled the gold set. ## Live leaderboard (Kodus Cloud) Next to the benchmark, the leaderboard has a live tab. Per model: precision as the share of suggestions that real teams applied on their own pull requests, as medians across Kodus Cloud organizations that review with their own model key, 2026-08-24 to 2026-11-15. A model is listed with 50+ PRs reviewed and when at least 2 organizations received 10+ of its suggestions. Self-hosted installs are not included. - Claude Opus 5.5: median 57.1% of suggestions applied, 1.5 suggestions per PR, 2,500+ PRs reviewed. - GPT-6.1 Sol: median 55.5% of suggestions applied, 1.15 suggestions per PR, 1,000+ PRs reviewed. - GLM-5.3 Flash: median 52.6% of suggestions applied, 1.43 suggestions per PR, 2,500+ PRs reviewed. - GPT-5.6 Sol: median 50% of suggestions applied, 1.05 suggestions per PR, 500+ PRs reviewed. - Gemini 3.8 Flash: median 49.9% of suggestions applied, 0.3 suggestions per PR, 1,000+ PRs reviewed. - Kimi K2.7 Code: median 47.9% of suggestions applied, 2.08 suggestions per PR, 1,000+ PRs reviewed. - Claude Haiku 5.5: median 47.2% of suggestions applied, 1.73 suggestions per PR, 500+ PRs reviewed. - GPT-6 Luna: median 47.1% of suggestions applied, 1.93 suggestions per PR, 1,000+ PRs reviewed. - Claude Sonnet 5.5: median 42.2% of suggestions applied, 1 suggestions per PR, 1,000+ PRs reviewed. - DeepSeek V4 Flash: median 42.2% of suggestions applied, 1.21 suggestions per PR, 500+ PRs reviewed. - GLM-5.3: median 41.7% of suggestions applied, 1.5 suggestions per PR, 500+ PRs reviewed. - DeepSeek V4.1 Flash: median 37.3% of suggestions applied, 2.06 suggestions per PR, 5,000+ PRs reviewed. - MiniMax M3: median 35.7% of suggestions applied, 3.02 suggestions per PR, 1,000+ PRs reviewed. - GLM-4.7 Flash: median 35.2% of suggestions applied, 1.21 suggestions per PR, 500+ PRs reviewed. - Gemini 3.1 Pro: median 32.8% of suggestions applied, 2.81 suggestions per PR, 1,000+ PRs reviewed. - GPT-5.4: median 62.7% of suggestions applied, 1.18 suggestions per PR, 100+ PRs reviewed; small sample, unranked. - GPT-5.5: median 62.1% of suggestions applied, 0.87 suggestions per PR, 100+ PRs reviewed; small sample, unranked. - GPT-6 Sol: median 52% of suggestions applied, 2.68 suggestions per PR, 250+ PRs reviewed; small sample, unranked. - GPT-5.4 mini: median 44.4% of suggestions applied, 0.76 suggestions per PR, 50+ PRs reviewed; small sample, unranked. - Claude Sonnet 4.6: median 44% of suggestions applied, 1.31 suggestions per PR, 100+ PRs reviewed; small sample, unranked. - Kimi K2.6: median 43.9% of suggestions applied, 1.93 suggestions per PR, 100+ PRs reviewed; small sample, unranked. - DeepSeek V4 Pro: median 42.9% of suggestions applied, 2.47 suggestions per PR, 100+ PRs reviewed; small sample, unranked. - GPT-5.6 Terra: median 34.2% of suggestions applied, 3.62 suggestions per PR, 250+ PRs reviewed; small sample, unranked. - GPT-5.6 Luna: median 33.7% of suggestions applied, 2.75 suggestions per PR, 100+ PRs reviewed; small sample, unranked. - GPT-4.1 mini: median 26.8% of suggestions applied, 0.7 suggestions per PR, 100+ PRs reviewed; small sample, unranked. - Gemini 3.7 Flash: median 24.7% of suggestions applied, 0.33 suggestions per PR, 100+ PRs reviewed; small sample, unranked. - GPT-4o mini: median 9.9% of suggestions applied, 4.17 suggestions per PR, 50+ PRs reviewed; small sample, unranked. Teams choose their model and differ in codebase and rules, so these rows are not a controlled comparison, and the live tab has no recall yet. The applied check runs on each organization's own model. ## Canonical sources - [Machine-readable leaderboard data](https://codereviewbench.com/api/leaderboard.json): metrics per model and mode. Prefer this or a model page for numerical claims. - [Leaderboard](https://codereviewbench.com/leaderboard): balanced mode; [heavy mode](https://codereviewbench.com/leaderboard?mode=heavy). - [Compare two models](https://codereviewbench.com/compare): side by side within one mode. - [Live leaderboard](https://codereviewbench.com/leaderboard?board=live): share of each model's suggestions applied on real pull requests; [as JSON](https://codereviewbench.com/api/live.json). - [Blog and analysis](https://codereviewbench.com/blog): posts written on the earlier v1 benchmark (Aug–Sep 2026); their numbers predate the current gold set. - [XML sitemap](https://codereviewbench.com/sitemap.xml). - [Public repository](https://github.com/kodustech/codereviewbench): scorecards, submissions, pipeline. - [Full site content](https://codereviewbench.com/llms-full.txt). ## Writing (v1 numbers) - [When an AI reviewer misses a bug, the code was usually right there](https://codereviewbench.com/blog/what-ai-reviewers-miss): We classified 792 missed bugs across thirteen models. 79% were real misses, where the relevant code was available and the model did not find the problem. Only 5% were artifacts of the test setup. - [One model on our board is right 74% of the time and finds 12% of the bugs](https://codereviewbench.com/blog/precision-and-recall-are-a-posture): Gemini 3.7 Flash has the best precision we measured and the worst recall. That is not a defect, it is a posture, and which posture you want depends on what the review is for. - [The cheapest model on the board finds more bugs per dollar than the most expensive](https://codereviewbench.com/blog/paying-more-does-not-buy-recall): Cost per pull request ranges from $0.08 to $1.62 across thirteen models. Recall does not follow. The cheapest option costs $0.07 per bug found; the most expensive costs $1.25. - [A three point gap between two AI reviewers is probably noise](https://codereviewbench.com/blog/one-run-is-not-a-measurement): Every entry on this leaderboard is one run. We measured how much the score moves from sampling and from the judge alone, and the answer is enough to reorder the middle of the table. - [AI reviewers are four times worse at Java than at Go](https://codereviewbench.com/blog/java-is-the-hardest-language): The same thirteen models, the same harness, the same scoring. Mean recall on Go pull requests is 59.1%. On Java it is 15.1%. Every model shows the gap. - [The best AI code reviewer finds fewer than half the bugs](https://codereviewbench.com/blog/ai-code-review-recall-2026): We ran thirteen models over the same thirty merged pull requests, scoring them against bugs that human reviewers actually flagged. The top scorer found 44% of them. None broke 50%. ## Citation Name the model, the review mode, the metric and the data date. Example: "Kimi K3.1 Pro (balanced mode) scored F2 63.5, recall 68.6%, finding 83/121 confirmed bugs on CodeReviewBench (2026-11-09)."