Test Questions in the Training Data: Even the Most Open AI Model Had Seen the Exam

Open Athena trained its 535-billion-parameter model Marin Mochi more openly than anyone before. Even so, test questions from common benchmarks slipped into the training data unnoticed. Using Marin as a case study, we explain how these leaks happen, why standard filters miss them, and which benchmark numbers you can still trust.

Analysis Nov 22, 2026 9 min read Original analysis by our editors
Illustrative image, AI-generated

What this is about

Few AI projects work as openly as Marin. The project comes out of Stanford CRFM and is now part of the nonprofit Open Athena. Starting August 20, the researchers ran the training of Marin "Mochi" 535B-A23B in public: loss curves, data mix, debugging and decisions were all on GitHub and Weights & Biases. Mochi is a mixture-of-experts model, where only a small part of the network does the computing for each token. It has 535 billion parameters, about 23 billion of them active. Pretraining wrapped up in early November after around 18.2 trillion tokens. On November 17, all 57 intermediate checkpoints followed on Hugging Face. According to project lead David Hall, that's around 61 TB in total, released under the OpenMDW 1.1 license.

There are still no benchmark numbers, though. The release blog post mentions the key point almost in passing. The trigger, it says, was "unexpected changes in eval loss" and concerns raised by Stanford student Michael Ryan:

"We discovered significant leakage from common pretraining benchmarks that our decontamination process had missed."

That affects which results can be fairly compared with other models. This article reconstructs what happened from the public GitHub issues. It also asks the obvious follow-up question: if this happens even with fully open data, what do benchmark tables tell you about models whose data nobody can inspect?

Three leaks, three mechanisms

1. Penn Treebank as word salad. On October 7, a team member noticed that eval loss on the Penn Treebank (PTB) portion of the Paloma test suite had dropped by about 0.2 nats. Loss measures how surprised a model is by a text. For this fairly niche material, the sudden drop was suspicious. The team rebuilt the exact training data stream and searched around 554 billion tokens. That turned up the GitHub repo quanpn90/torch-nplm, a reimplementation of Bengio's 2003 neural language model. About 99.6 percent of its roughly 1.39 million tokens consist of the complete PTB test and validation data, chopped into sliding six-word windows. On one test sentence, the checkpoint under review predicted the next token with 80 percent confidence; the following checkpoint, with just 0.2 percent. The team concluded the drop was "not a capability change" but came from verbatim exposure to the test text. It also predicted that the curve would drop again on the second pass through the data, around step 362,300. That's exactly what happened.

2. WikiText from the code dataset. An outside contributor (GitHub handle "bethehand") analyzed the public curves. They found similar one-sided dips on WikiText-103, while general Wikipedia text stayed stable. A search through all 8,192 files of the Stack v3 code dataset turned up the repo shaochenze/PatchTrain. It contained almost the entire WikiText test set. According to issue #9874, PTB and WikiText together accounted for most of the fluctuation in the Paloma average.

3. GSM8K may never have been checked. The most sensitive finding is in the data team's weekly report from October 20. In the Mochi run, 118 test tasks from lm-eval-harness failed to load because their dataset names had changed. Affected tasks included GSM8K, TruthfulQA, Winogrande, BoolQ and TriviaQA. The pipeline silently skips such tasks instead of failing. So these benchmarks were probably never used for decontamination. The report notes that this hasn't yet been confirmed against the data copy actually used.

This isn't a new problem for Marin. With the 32B predecessor, the team found GSM8K test problems in a cached data bundle and threw out an entire 1.07-trillion-token cooldown phase. And in a July audit, NVIDIA's open math corpus Nemotron-CC-Math contained text matching 141 of the 500 MATH-500 test problems (28.2 percent) and 17 of the 30 AIME 2024 problems. That shows how riddled web-sourced math data is with benchmark copies.

Why the filters fail

Marin's process is industry standard. It breaks every paragraph into 13-grams, sequences of 13 consecutive words, and matches them against the test tasks. It uses a Bloom filter for this, a space-efficient data structure that can say very quickly: "definitely not seen" or "probably seen." If at least half of a paragraph's 13-grams match test material, the document gets dropped. According to the data blog, this removed 13.66 billion tokens. The post itself admits the rule favors precision and therefore misses paraphrases and embedded examples.

Path into training Example at Marin Why the filter misses it
Fragmented copy PTB as 6-word windows in torch-nplm No 13 consecutive words, overlap 0.000
Short lines Answer options from MMLU/ARC, tables Paragraphs under 13 words are ignored, hit rate 0%
Embedded in other text Test question in the middle of a paragraph Below the 50 percent threshold, only 29.7% caught
Test set not on the check list Paloma, WikiText deliberately excluded, GSM8K probably not loaded What isn't checked can't be found
Paraphrase, translation general problem Word sequences no longer match

The numbers in the table come from issue #6852. There, a developer deliberately planted test questions in documents. Overall, the filter caught only 58.8 percent of them. A fix raised that to 83.3 percent, but for embedded questions it stayed weak at 37 percent. Using the raw test text instead of the windowed version, the PTB match came out at 1.000, by the way. So the filter works, but only for the form it expects. Stricter rules come at a cost: according to the weekly report, the old rule would have removed around 87 billion tokens, the new one around 8 billion. Most of the old rule's hits were harmless short lines.

Paraphrases are the older problem. In 2023, LMSYS researchers showed that a 13B model trained on rephrased test questions scored 85.9 on MMLU, GPT-4 territory, without tripping an n-gram filter. In RedPajama and StarCoder-Data, 8 to 18 percent of HumanEval overlapped with the training data. In 2026, a paper using embedding search, which matches by meaning rather than wording, found semantic duplicates of 100 percent of MBPP and 77.5 percent of CodeForces problems in the training corpus of AI2's Olmo 3. And that was after decontamination.

How to spot contamination

  • Jumps in loss: The Marin case is a textbook example. Smooth curves that suddenly drop on exactly one test set are a red flag. On the Uncheatable Eval test sets, made of text written after the data cutoff, Marin showed no such drops.
  • Freshly written parallel tests: In 2024, Scale AI commissioned GSM1k, 1,205 new math problems in the style of GSM8K. Some models scored up to 8 points lower on it, especially the Phi and Mistral families. The more likely a model was to generate GSM8K examples verbatim, the bigger the gap. Frontier models showed little sign of overfitting.
  • Time cutoffs: LiveCodeBench sorts problems by release date. A 33B DeepSeek base model dropped from around 60 to around 0 percent on LeetCode problems published after August 2023.
  • Probability tests: Methods like Min-K% Prob check whether a text contains suspiciously few "surprising" words. That suggests the model already knows it.
  • Probing: In February 2026, OpenAI showed that frontier models could reproduce the original solutions to SWE-bench Verified tasks. Gemini 3 Flash managed it from the task ID alone. OpenAI has stopped reporting the benchmark since.

The question hardly anyone asks

Marin could find its leak because the data, data stream and curves are public and outsiders could help search. With closed models, none of those paths exist. Model Cards usually just describe the procedure, such as 13-gram matching for Qwen 2 or 8-grams for Llama 3. For Llama 3, Meta itself wrote that so many examples in MBPP, HumanEval, MMLU and MMLU-Pro were flagged as overlapping that the performance gain couldn't be reliably estimated. For Llama 1, researchers using the ConTAM method estimated the contamination advantage at around 18 points on HumanEval and 25 on Big Bench Hard. Even the exemplary open K2 Horizon from Abu Dhabi, according to its paper, decontaminated only the 18 benchmarks it reports on, out of a registry of 208. Paraphrases wouldn't be caught.

This week shows what that looks like in practice. On the Claw-Eval leaderboard on Hugging Face, the top seven spots come exclusively from vendor-reported numbers, and none of the 22 entries is verified. GPT-6.2 Astra scores 100 percent on ARC-AGI-3 but only ranks 7th on the Vals Index and, at 77.3 percent on SimpleBench, scores below its predecessor. That doesn't prove contamination. But it does show how far public and hidden measurements can diverge. According to the project tracker, Percy Liang warned back in September during a model comparison that the compared models may have trained on the test sets, so the gaps were a lower bound.

Which numbers are more robust

Type Examples Limitations
Hidden test sets Vals Index, SimpleBench (only 10 sample questions public), ARC Prize private set "Semi-private" sets tested via APIs can leak over time, according to ARC Prize
Newer than the data cutoff LiveBench (refreshed monthly), LiveCodeBench, Uncheatable Eval Only clean until the next model generation has read them
Parallel tests GSM1k Costly to build, available for only a few benchmarks
Vendor numbers on well-known tests MMLU, GSM8K, HumanEval Most vulnerable, data can't be verified

Our assessment

Our take: the Marin case isn't a scandal; it's a stroke of luck for research. A team that discloses everything shows that standard filters have big gaps even with the best intentions. The causes are mundane: a fragmented copy in a GitHub repo, a deliberately excluded test suite, a silent loading error. What the team hasn't said yet is also notable. It hasn't named all the affected benchmarks or quantified the effect in points. Whether GSM8K was actually in the training data isn't confirmed in the public records. The irony: GSM8K and MATH-500 are among the first tests planned for Mochi (issue #11869).

What this means for you: vendor numbers on old, public benchmarks are a rough indication at best. Give more weight to results on hidden or demonstrably newer test sets measured by independent parties. No test is completely immune. But only with open data like Marin's can you even check how big the problem is.

Sources

#benchmarks#contamination#marin#open athena#open source#evaluation

Ask the AI assistant

Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.

Connecting …

The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.

Comments

What do you think? Already tried it? Write it here, no sign-up needed.

  1. Loading comments …

Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.