63% of AI benchmarks 'solved'? Count only independent runs and it's 24%

What this is about

A benchmark is a standardized test used to compare AI models, such as a set of questions or a collection of coding tasks. Once a benchmark is "saturated," the best models get almost everything right, and the test can no longer tell good from very good. On November 14, 2026, Ethan Mollick, a professor at the Wharton School, published SatBench, an analysis that measures how fast this happens. His headline on X:

"AI is getting better faster than our instruments for measuring it."

According to SatBench, 63 percent of 173 benchmarks are solved. Further down the README, in the caveats section, there's a second number: 24 percent if you only count independent measurements. That gap of almost 40 points is what this article is about. We looked at specific benchmarks to see what's behind it. A Chinese summary on rss.qiaomu.ai also mentions both numbers but doesn't dig into individual cases.

What SatBench measures

SatBench treats the 173 benchmarks that AI labs cite most often as one big exam. A test counts as solved once the best reported score reaches 85 percent. The data comes from Brian Zhou's Benchmark Almanac, an open database that says it tracks 3,327 benchmarks, 1,170 models and 104 labs. It collects scores from launch posts (the announcements labs publish with new models), system cards, official leaderboards and Epoch AI's Benchmarking Hub. Results from Artificial Analysis are excluded because of its terms of use.

The key findings, according to the README:

• 109 of 173 benchmarks are above 85 percent. From 2023 to early 2025, the index hovered around 30 percent, because new tests were added about as fast as old ones were solved. Over the past twelve months, it climbed by roughly 20 points.
• Each year, a bigger share of new benchmarks was solved within twelve months: 17 percent for the 2023 cohort, 30 percent for 2024, 43 percent for 2025. The median time to 85 percent fell from about 20 months to about 14.
• OpenAI accounts for the most first crossings ("kills") on tests reported by several major labs (20). In 2026, Anthropic led (11).
• The number depends heavily on the definition. With a 75 percent threshold, 82 percent would count as solved. At 95 percent, only 25 percent would.

According to the AI Watch newsletter, Mollick had Claude Fable 5.2 build the analysis in about four prompts. He calls the projection to 2028 and 2030 an extrapolation and explicitly not a forecast.

The real finding: three out of four records are self-reported

The most important sentences are in the detailed summary. 75 of the first 100 crossings come from the lab's own launch post, "a lab grading its own model." Only 10 of those 75 were later confirmed by an independent run at 85 percent or above. For 44 benchmarks, there's no independent run at all. For the remaining 129, the index stands at 33 percent with independent runs only and 60 percent with all sources.

The summary doesn't say which ten records were confirmed. You'd have to pull them from the raw data using the scripts. So we checked individual benchmarks in the Almanac:

Source: Benchmark Almanac as of November 14, 2026, vals.ai. Our own compilation.

The table shows three patterns. On classic exam-style questions like GPQA Diamond, labs and independent testers agree. In an analysis from December 2025, Epoch AI explicitly states that lab-reported GPQA Diamond scores match its own runs. So the test really is solved, yet according to SatBench it still showed up in 102 launch posts in 2026, all of them after the 85 percent mark had been crossed.

With GDP.pdf and Chartography, on the other hand, the "kill" exists only in the launch post. Anthropic's 85.4 percent on GDP.pdf was measured without tools and graded by Claude Opus 4.7, meaning an in-house model acted as the judge. On the official leaderboard, the same model scores 49.4 percent. Chartography is even more extreme: DeepSeek V4.1 Flash reports 78.9 percent with tools, while the leaderboard shows 11.6 percent.

On agent benchmarks like Agents' Last Exam, the lab's number is higher than the leaderboard number in all six mismatched pairs. GPT-6 Astra scores 59.3 percent according to OpenAI and 34.2 percent according to the leaderboard. GPT-6 Luna reports 50.9 percent, while the leaderboard shows 25 percent.

Why the numbers diverge

Mostly, nobody is lying. The labs pick the conditions themselves. The biggest lever is the harness, or scaffold: the software around the model, with its prompts, tools, time limits and retries. According to Epoch AI, switching scaffolds alone moves SWE-bench Verified results by up to 11 points (GPT-5) or 15 points (Kimi K2 Thinking). On top of that:

• Tools: According to the Almanac, Humanity's Last Exam "with tools" is its own reporting mode. Leaderboards from different providers differ by up to 20 points here.
• Different metrics: On OSWorld 2.0, labs report partial credit, while the official leaderboard only counts fully completed tasks. That's how Meta Muse Spark 1.3 ends up at 66.9 and 32.0 percent respectively.
• Subsets: GPT-5.4's 98.9 percent on τ²-bench applies only to the telecom domain.
• Versions: Anthropic cites 92.1 percent on Terminal-Bench 2.0 for Mythos, but "with the fixes from 2.1 and a four-hour time limit." The Almanac doesn't count that as solved.

The gap doesn't always favor the labs. On Terminal-Bench 2.0, the leaderboard measures 84.7 percent for GPT-5.5, while OpenAI itself reported 82.7 percent. On the security test CyberGym, lab-reported scores sit a median 12.7 points below the leaderboard. Across all 1,315 score pairs on the Almanac's comparison page, 461 of 726 discrepancies favor the lab. But the median is just 1.4 points. The big gaps are concentrated in a handful of benchmarks: Chartography (+37.9), Cybench (+24.1), Agents' Last Exam (+23.6), MRCR v2 (+15.8).

Why hardly anyone re-runs the tests

Independent measurements are expensive and arrive late. In April, the EvalEval Coalition compiled costs on Hugging Face: a single run of SWE-bench Verified Mini costs between $4 and $1,600 depending on model and harness, and GAIA up to $2,829. The Holistic Agent Leaderboard costs around $40,000 in total, or about $320,000 with eight repetitions for solid statistics. The authors warn:

"Whoever can pay for the evaluation writes the leaderboard."

Then there's access. According to its methodology page, Epoch AI only reports models it can run itself. Claude Mythos Preview, the first to "kill" SWE-bench Verified, was only available to select partners. Some testing setups have also gone quiet. Epoch hasn't run SWE-bench Verified since June 2026, and vals.ai archived its leaderboard on September 1. OpenAI stopped reporting the test back in February anyway: its own review found that, by Epoch's estimate, at least 16.4 percent of the tasks are flawed. Epoch therefore rates the benchmark as "Flawed."

Our request to Epoch AI and Brian Zhou asking why so few records get re-measured had gone unanswered by press time.

"Independent" isn't clear-cut either

SWE-bench Verified exposes another weakness of the 24 percent figure. With its simple scaffold, Epoch measures a best score of 83.5 percent (Claude Opus 4.7). Using the mini-SWE-agent harness, vals.ai measures 97 percent for Claude Opus 5 and has 16 more entries above 85 percent. Both are third-party measurements, and they're 13 points apart. The Almanac lists only the 83.5 percent as the best independent score. So which testers count as "independent" helps determine how high the 24 percent ends up. Our take: That figure is more of a floor, and the 63 percent more of a ceiling.

SatBench itself also warns about errors in the data. Some records are dated before the test in question was even published. Qwen3-30B, for example, is supposedly the first to cross 85 percent on AIME 2026, in July 2025. For Thinking Machines' Inkling 2 Small, the Almanac entry for APEX-Accounting (14.5 percent) contradicts the lab's own figure (62.0 percent).

A cross-check from research

An independent study reaches a similar overall picture using a different method. The ICML paper "When AI Benchmarks Plateau" by Mubashara Akhtar and 36 co-authors examines 60 benchmarks. There, saturated means the top models can no longer be told apart statistically. There's no fixed percentage threshold. The result: 29 of 60 benchmarks, or about 48 percent, are heavily saturated. For tests no older than 24 months, it's 42.9 percent; for tests older than 60 months, 54.5 percent. The authors say that difference isn't statistically significant. Private test sets don't prevent saturation, while curation by domain experts tends to slow it down. That fits Mollick's follow-up: "Private benchmarks start harder but fall just as fast."

Our verdict

Today, "benchmark saturated" can mean three different things. First: the models really have gotten that good, and independent testers confirm it, as with GPQA Diamond or Terminal-Bench 2.1. Second: a lab reported 85 percent under conditions it chose itself, and nobody re-measured, as with GDP.pdf or Chartography. Third: the test itself is broken or contaminated, meaning the solutions leaked into training data, as with SWE-bench Verified.

From everything the independent data shows, the acceleration SatBench documents is real. On classic exam questions, lab and third-party scores barely differ. But for agent, tool-use and document tests, the launch post is often the only evidence. That includes this week's records. LongCat-2.5 reports 88.4 percent on Terminal-Bench 2.1 and DeepSeek V4.1 Flash 90.6 percent, and neither has a leaderboard entry. Gemini 4 Hydrogen reports 58.8 percent on HLE-Diamond, again with no leaderboard entry. At the other extreme is Qwen4-Max: according to reports, Alibaba published no benchmark scores at all with the preview on November 11.

Our take: When you see a new record, check who measured it, with which harness, and whether there's a leaderboard entry. If the number only appears in a launch post, it's a vendor claim, not proof. What independent testers need most is money and access.