AI peer reviewers find most planted errors – but omissions remain their blind spot

What this is about

Measuring how well an AI reviews scientific papers is hard, because with real papers nobody knows exactly which errors they contain. Several research groups solve this problem the same way: they deliberately plant errors in real papers. The technical term is error injection. Then they count how many of those errors an AI reviewer finds. This November, two such benchmarks were released along with open data. One, by Íñigo Zubeldia and colleagues (Cambridge, Flatiron Institute), tests astrophysics and ML preprints; the other, by Paul Litvak, tests psychology papers. We compared them with earlier work.

Our thesis up front: AI review works well as a second pair of eyes, ideally with several systems. But the headline number "AI finds X percent of errors" depends heavily on the test setup. During our research, not every widely circulated claim held up. That belongs in this article too.

The principle: plant errors, count hits

Zubeldia's team planted four types of errors: citations, math, computed values and general inconsistencies, each at three severity levels from subtle to obvious. All papers were published after the knowledge cutoff of the models tested. Twelve systems were evaluated. They include Claude Opus 4.7, GPT-5.5 and Gemini 3.1 Pro with a single call, two variants of Claude Code with Python and computer algebra (software that checks formulas symbolically), and the Stanford Agentic Reviewer. Added to that are two fine-tuned open models and Skepthical, the authors' own multi-agent system. Skepthical downloads cited papers, re-derives formulas with SymPy and checks numbers with code.

Selected results (overall recall, astrophysics / ML):

The 91.1 percent figure quoted in news reports is accurate. But it only applies to a slice of the data: calculation errors in astrophysics papers, found by Claude Code with a structured prompt. For citation errors, the same configuration managed 26.7 percent. The two open models, trained on old conference reviews, found less than 3 percent. The authors put it this way:

"Conversational training on past conference reviews does not equip models to localize technical errors."

The authors acknowledge the limitations themselves. Each corpus consists of 15 papers from the same day. Hits are judged by a panel of language models, and a human checked only one paper. Skepthical also gets more calls than the single models. Against GPT-5.5 with a short structured prompt, it comes out only 1.7 and 0.6 points ahead, and that difference isn't statistically significant.

Finding 1: The prompt matters a lot – the model matters even more

The +32.2 and +38.9 percentage point figures are correct. They apply to Claude Code with the same model (Opus 4.7) and the same tools, once with an open prompt and once with a structured prompt that lists the error categories. With the open prompt, the agent didn't write a single verification script. With the structured prompt, it wrote 21 scripts in the astrophysics run. So the prompt decides whether the tools get used at all.

The popular reading that the prompt moves the needle more than switching models, however, isn't supported by the paper. Given the same open prompt, the three frontier models are 48.6 and 52.8 points apart. The authors explicitly call model choice the biggest factor. The prompt effect also varies widely: Gemini gains 40 points on the ML corpus, while GPT-5.5 gains just 0.8 in astrophysics. In Litvak's study, the category prompt gave GPT-5.5 only 5 additional hits (66 instead of 71), GPT-5.4 gained 13, and Claude 4.6 and Gemini gained 0 to 3.

Our take: a single percentage for "AI review" means little without specifying the model, prompt and tools. Even small changes to the setup shift it by dozens of points.

Finding 2: Multiple systems find more, with diminishing returns

In Litvak's study, GPT-5.5 alone finds 71 of 100 errors. Add the commercial tool Refine.ink and it's 80; add a third system from a different model family and it's 85. Five systems reach 90, eight reach 93, and the remaining six configurations add nothing. In the Agentic Review benchmark from Chenhao Tan's group (Chicago), recall rises from 71.6 percent with the best single model to 83.3 percent with all of them combined. Zubeldia's team doesn't report a combined rate for all systems. It does show, however, that Skepthical and Claude Code hit citations and numbers almost independently of each other (correlation r ≈ 0.04).

The costs differ enormously. According to Litvak's analysis, each error found with GPT-5.5 costs about 4 cents. Refine.ink costs around $50 per paper, or roughly $8.77 per error – more than 200 times as much. In return, it finds most of the errors no other system catches. In Zubeldia's study, Skepthical costs $2.41 per paper and Claude Code with a structured prompt around $1.73.

There's counter-evidence, too. A study in which 45 experts rated reviews of 82 papers from the Nature family of journals (2605.20668) found that AI reviewers overlap with each other far more than humans do (21 versus 3 percent). The "Science Prediction Universe" study by UniPat AI (2611.15049) shows that AI agents agree more with each other on predictions (0.52) than with lab measurements (0.33). For one cancer study, eleven configurations agreed at 0.90 but matched reality at only 0.27. Combining several systems only cancels out the mistakes they don't share.

Finding 3: What's missing from the text is what systems find worst

Litvak didn't just plant false statements – he also deleted information. Examples include a statement about preregistration (the study plan published in advance), an exclusion criterion for participants and a calculation step in data processing. According to the paper's summary, all seven errors that no system found were omissions of this kind.

But the claim that systems fundamentally can't detect omissions isn't true. According to Litvak's detailed analysis, a single system finds on average 33 percent of the 24 deleted passages, and GPT-5.5 alone finds 13 of them. All systems combined reach 83 percent. The analysis also contradicts itself: in one place it says four of the deleted passages went undetected, elsewhere it's all seven. What's certain: omissions are the weakest category. For comparison, all systems combined found 100 percent of inconsistencies within the immediate context of the text. Litvak considers omissions the most dangerous errors, precisely because results often fall apart on them.

Finding 4: Hardly anyone measures false alarms

Litvak and the Agentic Review benchmark count only hits. How often a system flags an error that isn't actually one, they don't check. Litvak writes that 84 percent of the flags concerned things he hadn't planted. He couldn't verify most of them. In the Agentic Review benchmark, users submitted real feedback. There, false alarms were the most common reason for negative ratings, at 38.5 percent.

Skepthical is the only benchmark that measures this systematically, using unmodified copies of the papers. Most systems stay below 10 percent false alarms there. But 30 percent (astrophysics) and 67 percent (ML) of math-related flags were wrong. With the short prompt, the false alarm rate of the single models rose to 17 to 27 percent. If you only read the recall figure, you're seeing half the truth at best.

In practice: conferences use AI and ban it at the same time

Conferences and publishers are responding inconsistently. At AAAI-26, all 22,977 submissions received an additional, labeled AI review. In the follow-up survey, 53.9 percent found it useful. But 49.4 percent of reviewers also said the AI had missed points that humans would have caught. ICML, by contrast, desk-rejected 497 papers (around 2 percent) without review. The reason: 398 reviewers who had also submitted their own papers had used AI despite having opted into a ban themselves. They were caught through hidden instructions in the PDFs. For ICLR 2026, detector vendor Pangram estimates that 21 percent of the roughly 75,800 reviews were fully AI-generated. That's one vendor's analysis, and nobody has independently verified it. According to 2603.20450, AI detectors also often classify reviews that were merely polished linguistically with AI as fully AI-generated.

In Germany, the DFG (German Research Foundation) has permitted AI in peer review since April 16, 2026. Conditions apply: AI use must be disclosed, only secured systems are allowed, and responsibility stays with the human. Elsevier prohibits reviewers from uploading manuscripts to AI tools, but says it's working on AI features for its own editorial systems. Since 2025, Springer Nature has used an in-house tool that detects irrelevant references. The American Economic Association runs technical checks on papers in the revision stage with Refine.ink.

The same question is coming up in software development. In the Linux kernel, developer Lorenzo Stoakes is stepping back from reviewer roles because AI-generated patches are flooding his queue. Anthropic describes a tiered process for Claude Code: several AI reviewers check every change, and depending on the risk, zero, one or two humans must approve.

Our assessment

Our take: the benchmarks show that AI reviewers find a large share of objectively verifiable errors – especially calculation errors and inconsistencies – when they use tools. A combination of different systems demonstrably finds more. But on closer inspection, three widely circulated claims turned out to be inaccurate:

• The prompt matters, but the model usually matters even more.
• Omissions are the weakest category, but they aren't missed without exception.
• False alarms aren't a side issue: in math in particular, many flags are wrong.

What remains open is how well planted errors reflect real ones, which tend to be more subtle and often concern novelty or study design. For comparison, a checking tool from James Zou's group found errors in real AI papers, and experts confirmed 83.2 percent of those flags. For authors, the practical advice is cheap to follow: run your paper through two or three models from different vendors before submitting. By Litvak's numbers, such a check costs just a few cents per error found. It doesn't replace peer review. An error the AI doesn't flag can still exist, and a flagged error isn't automatically real.