Proofs, Yes. Pipettes, No: Why AI Scientists Struggle in Real Labs

What's going on

Over the past few weeks, stories about "AI as a scientist" have arrived almost daily. Claude helped track down 663 theorems in Gödel's notebooks. An internal OpenAI model produced an existence proof for Yang-Mills theory. Edward Witten credited Claude as a source of ideas in a paper. On the TCSExam benchmark, GPT-6 Astra proved 14 of 20 research theorems.

At almost the same time, three benchmarks came out that complicate that picture. In each one, AI agents have to predict the outcome of real lab experiments or run those experiments themselves: OpenAI's BioBasicsBench (November 6), then AISLE shortly after, with 4,079 runs on lab robots, and Science Prediction Universe (SPU), posted to arXiv on November 11. All three reach a sobering conclusion.

Our thesis: AI is strong wherever a computer can check the result, as with proofs, code and simulations. When nature has to deliver the verdict, performance collapses. And the bottleneck isn't so much using tools as exercising scientific judgment.

The successes: there's always a checker in the room

Look at how the success stories were verified, and a pattern emerges:

Former OpenAI researcher Jason Wei put this into a rule back in July 2025: how easily you can train AI on a task depends on how easily the solution can be verified.

"All tasks that are possible to solve and easy to verify will be solved by AI."

Wei's criteria are an objective truth, verification in seconds, many checks running in parallel, low noise, and the ability to rank solutions by quality. A Lean proof meets nearly all of them. A lab experiment meets hardly any: it takes hours or days, uses up materials and produces noisy results.

Even in math, verification isn't free. Google writes that humans first have to confirm that the formal statement actually matches the original problem. In the Yang-Mills repository, that exact question of whether the translation is faithful remains open, and the mass gap is explicitly outside the scope of the proof.

Test 1: Science Prediction Universe – more compute doesn't help

SPU comes from a team led by UniPat AI. Its 116 tasks span chemistry, materials science, biology and medicine. 100 of them are based on 85 studies published in Nature journals in 2025 and 2026, and another 16 on unpublished measurement series from partner labs, covering things like nanobodies and electrolytes. The agents get a set of experiments and have to rank them by a measured but hidden outcome, such as yield or conductivity. Scoring uses Spearman correlation, a measure of how closely a ranking matches the true one. A value of 1 is perfect, 0 is chance. Each agent had eight hours and one GPU.

The authors see two performance tiers, and within each tier the ranking can't be resolved statistically. The configurations beat chance on only 34 to 52 percent of their scored tasks. The top score also comes with a caveat. Opus 5.5 refused 56 percent of the public tasks, and Fable 5.1 refused 69 percent. All 125 refusals carried the label of the provider's biosafety filter. Both models turned down every medical task.

The clearest finding is about compute. At its highest reasoning level ("Max"), GPT-6 Astra ran 4.8x as long and cost 5.3x as much as at medium (median figures), averaging $65.91 per run instead of $9.59. The results didn't get any better. Medium was actually slightly ahead, though within the statistical margin of error. Runtime, number of tool calls, web fetches and GPU usage also showed no clear link to the results. The authors highlight the contrast with software benchmarks, where more compute reliably helps.

The error analysis shows why. The agents calculate proxies, stand-in values that are easy to compute, such as binding energies or docking scores. They don't check whether those values actually drive the measured outcome. On a task about luminescent boron compounds, runs that ranked by the strength of the optical transition scored between −0.50 and 0.17. Runs that looked at the loss channels instead reached 0.55 to 0.88. A quarter of the submitted rankings simply follow a single number like temperature, while only 11 percent of the real measurements do. Later corrections didn't help on average. The agents expected a correlation of 0.61 and achieved 0.49, so their self-assessment told you very little about how they actually did. And they line up more with each other (0.52) than with the experiment (0.33). The authors conclude:

"The bottleneck lies in scientific judgment, not in tool execution."

Test 2: AISLE – real robots, zero percent on safety

AISLE ("AI Scientists' Last Exam") was started by Tong Xie, founder of the startup Rhythm Creation. Participants include UNSW Sydney, NUS Singapore, EPFL and Berkeley, along with researchers from Nvidia, Microsoft Research and Eli Lilly. Seven models controlled two automated labs: the model proposes a formulation, the robot mixes it, and sensors report the measurements back. Each site has a digital twin, a simulation of the lab where ideas can be tested in advance. In total, the project logged 4,079 experiments and 46,626 minutes of instrument time. The leaderboard and code are available at aisle.science and on GitHub.

According to ifeng and 36Kr, the gaps are large:

• Simulation vs. reality: GPT-5.5 scored 90.74 points on optimization in the dry run, and 16.46 in the real lab.
• Color formulations: On the eight-component task, none of 54 runs hit the strict target. With four components, 3 of 54 runs succeeded in Sydney and 15 in Singapore.
• Drifting pipette: The researchers secretly tampered with the dosing. Only 12 of 20 runs made it back into the target window. Gemini failed all three cases. Claude noticed the change but kept mixing old and new data until the final round. Kimi dismissed the deviation as noise. According to 36Kr, no model asked for the instrument to be recalibrated.
• Viscous liquids: As the resin's viscosity increased, the pass rate fell from 86.9 to 32.9 percent.
• Safety: In the simulated safety section, the forbidden zones were known in advance. Even so, no model reached the target without a violation, and every model's pass rate was 0 percent. One GLM run judged a formulation safe twice in a row and broke the rules both times. GPT-6 Astra scored 96.53 points on dry-run optimization, but only 24.98 on safety.

GPT-5.5 reached its overall score of 41.19 points across all completed wet-lab disciplines. No single model led in every area. One important caveat: GPT-6 Astra and Opus 5.5 are still listed as "in progress" on the wet-lab leaderboard, so the models tested are roughly a generation old. According to GitHub, the paper itself is still an unpublished manuscript.

Test 3: BioBasicsBench – the friendliest result

OpenAI teamed up with enzyme startup Ligo to put together 100 everyday lab tasks: gel images, bacterial plates, microscopy. The benchmark only measures data analysis. Pipetting and operating instruments aren't part of it. GPT-6 Astra scored 66.7 percent of possible points, compared with 43.8 percent for GPT-5.6 Sol. That's a clear step forward. According to R&D World, Astra read a handwritten note on a control plate saying the wrong antibiotic had been used.

The failure examples still fit the overall picture. Astra ordered the gel bands correctly but then picked the wrong colonies, and on one plate it overcounted colonies by about 12 percent. The observation was right, the conclusion wasn't. OpenAI writes:

"Higher benchmark scores do not yet mean that models can make reliable judgments in the lab."

Some criticism remains: OpenAI only tested its own models, there's no human baseline, and according to Kingy AI the tasks haven't been published.

The pattern behind it

Earlier studies back up the finding. In Scale AI's SciPredict benchmark (April 2026), 15 models correctly predicted experimental outcomes in 14 to 26 percent of cases, and domain experts in about 20 percent. The difference lies in calibration, meaning whether you know when you can be confident. For humans, the hit rate rose from about 5 to about 80 percent the more predictable they judged a question to be. For the models, it stayed around 20 percent regardless of how confident they were. A study led by Kevin Jablonka covering more than 25,000 agent runs found that evidence was ignored in 68 percent of runs. Agents revised their assumptions after a contradicting result in only 26 percent of cases. A Hacker News commenter who says he has 45 years of lab experience also points out that even humans at the bench differ in reliability by at least a factor of ten.

A fourth candidate, Science's Last Exam, doesn't count yet. According to its website, its wet-lab section is "under construction."

Our verdict

Our take: the wins in math and the failures in the lab don't contradict each other. They follow the same logic. Where a checker can score millions of attempts in seconds, a model can search, fail and try again until something works. In the lab, that feedback is missing, or it's expensive and noisy. Then what counts is whether the model understands which mechanism determines a measurement, and that rarely works yet. The fact that the highest reasoning level brings nothing on SPU fits this pattern. If you start from the wrong approach, more compute won't get you anywhere.

The lab benchmarks do have limits, though. They're small (16 private SPU tasks), some haven't been peer-reviewed yet, and AISLE is missing the newest models in the wet lab. Where the lab gives fast, cheap feedback, AI already helps today. According to ifeng, GPT-5 tested more than 36,000 formulations for cell-free protein production with Ginkgo Bioworks and cut costs by about 40 percent. At its core, that's once again a search with a verifier, except this time a robot does the checking.

For labs, the takeaway is this: AI agents are useful for analyzing data and generating ideas, but not as unsupervised experiment leads. That applies especially as long as no model passes the AISLE safety section.