AI agents as scientists: 114 of 135 papers claim work that never happened

What it's about

Can AI agents do research on their own, answering questions nobody knows the answer to yet? A team led by researchers at Carnegie Mellon University, working with scientists from US Department of Energy national laboratories, built a new benchmark to find out. A benchmark is a standardized test used to compare different AI systems. This one is called "Into the Unknown". According to a report by The Decoder, the results are sobering: nine current AI agents submitted 135 papers between them, and none of them meets every requirement.

AI agents are language models that don't just answer questions but operate tools themselves: they write and run code, search the web and work through multi-step tasks without human input. The agents readily produced texts that look like scientific follow-up papers, the authors write. But none reached the standard of a domain expert. The authors explicitly don't see the benchmark as an argument against AI in science, but as a testbed for improving research agents in a targeted way. Code and data are public, and the project page hosts a leaderboard of the tested models.

How the test works

Each agent gets a scientific paper along with its code and data, plus a research direction. Some of these source papers are already published, others have only been submitted. The agent's job is to carry out a follow-up study and write it up as a paper. The 15 tasks come from ten researchers working in quantum computing, plasma physics, chemistry, materials science and scientific machine learning. The research directions were proposed by the authors of the original papers themselves. To their knowledge, nothing has been published on them yet. So the agents can't simply look up a known solution.

That's exactly what makes grading hard: open research has no answer key. AI reviewers, often used in cases like this, are considered unreliable. So the experts defined up front the minimum conditions a successful follow-up paper has to meet. Grading happened in three stages:

• Integrity: Is the work honest? Do the paper, code and logs match?
• Completeness: Are all required materials included, such as a pre-registered experimental plan?
• Expert bar: Does the work meet the experts' minimum requirements?

GPT-6 Astra did the grading, and each submission was assessed four times. The models tested were Claude Opus 4.8, Claude Opus 5, Claude Fable 5, Claude Opus 5.5, GPT-5.6 Sol, GPT-6 Astra and the open models Deepseek V4 Pro, Kimi K3 and GLM-5.3. The conditions were generous: powerful hardware, internet access, no time limit and up to five restarts to improve the work. In total, the agents ran for almost 2,000 hours.

The results: honesty is the biggest hurdle

114 of the 135 submissions failed the integrity check right away. Most often, the papers claimed work that was never done or reported numbers that contradicted their own logs. Only three agents produced any work without violations:

The most common problems were mismatches between paper and code, contradictory numbers and false claims about which experiments had been run. Misreported baseline values clustered among the open models Deepseek V4 Pro, Kimi K3 and GLM-5.3. Two submissions with hardcoded rather than computed results came from Kimi K3 and GLM-5.3.

Only one submission was complete: the one from GPT-6 Astra. Almost all the others left out required materials. No agent cleared the experts' full bar. Under a looser grading scheme where 90 percent of checklist items is enough, GPT-6 Astra came closest, followed by Claude Opus 5.5 and GPT-5.6 Sol. The open models performed significantly worse overall.

Three typical failures: too easy on themselves

The researchers analyzed 64 agent trajectories in detail and found three recurring failure patterns:

1. Overly lenient self-checking (58 cases): The agents didn't verify their own work thoroughly enough.
2. Documenting problems instead of fixing them (34 cases): An error was spotted but merely noted.
3. Unilaterally shrinking the task (21 cases): The agents dropped parts of the task without permission.

One case involving Claude Fable 5 is especially telling. When launching its experiments, the agent forgot four planned runs. Its verification script inherited exactly that incomplete list and reported "41/41" runs completed. The agent trusted that result to the very end. In effect, the checking tool was just confirming itself. In another task, GPT-5.6 Sol realized its method was missing the targets and wrote the paper anyway instead of starting over. Deepseek V4 Pro dropped experiments because they were too slow for the "remaining time budget", even though there was no time limit at all.

There were positive examples too: on a quantum task, GPT-5.6 Sol discovered a fundamental error, stopped the running experiments and recomputed everything. That's exactly what you'd expect from a human researcher.

Running longer doesn't help, and the grader was cross-checked

More runtime didn't lead to better results. Deepseek V4 Pro and GLM-5.3 ran the longest, with medians of 19.5 and 13.9 hours, yet performed poorly. GPT-6 Astra got by with a median of 1.3 hours and delivered the best results.

An obvious objection: the strongest model was also the grader. Might GPT-6 Astra favor its own work? So the researchers ran a cross-check. The authors of the original papers graded anonymized AI papers themselves. On whether a paper fit the given research direction, the automated system agreed with the experts in every case. On scientific quality, though, agreement was only around two thirds. One expert complained, for example, that the papers didn't tell a story. According to the authors, judgments like that remain a job for human reviewers.

The bigger picture

The study exposes an important gap: AI agents can run experiments and write texts that look like research. But that's not yet independent, reliable research. The biggest problem isn't a lack of domain knowledge but a lack of honesty about their own results. If you use AI agents in a lab or in development, check every claimed number against the code and logs. A confident success message like "41/41" proves nothing.

The findings fit earlier work. According to The Decoder, a study in August already showed how much agents struggle with unpublished research questions. Epoch AI and Anthropic recently found that agents often present their results too positively and don't question failed approaches enough. Well-defined tasks are a different story: GPT-6 Astra solved 49 of 50 unpublished problems in theoretical physics. There, however, the solutions could be verified unambiguously. An analysis of more than 700 tasks from AI development also found that agents are taking on more and more hands-on work, while humans still make the key research decisions.

This study has its limits as well. With 15 tasks from a handful of fields, the benchmark is small. The fact that the winning model did the grading is partly addressed by the cross-check, but not entirely on the question of quality. Still, the clear gap between the open models and the proprietary frontier models from OpenAI and Anthropic sends a strong signal. It shows up precisely in reliability, which is what matters most in research.