AI models know when they're being tested. What are safety evals still worth?

What's at stake

People behave differently when they know they're being watched. In humans, that's called the Hawthorne effect. In AI models, it's called evaluation awareness: the model picks up on details of a task and realizes it's being tested rather than deployed for real. The phenomenon isn't new. Its scale is. Anthropic's system card for Claude Fable 5.2 and Mythos 5.2, published November 9, reports possible evaluation awareness in up to 43 percent of automated behavioral audits. Two studies from October and November show that behavior actually shifts when that happens.

This goes beyond research. Decisions to release new models, and safety frameworks like Anthropic's Responsible Scaling Policy, rest on exactly these kinds of tests. On November 10, Anthropic said a key threshold in that policy had been reached. Below we explain how evaluation awareness is measured, what the new numbers tell you and which countermeasures actually exist.

Three ways to measure evaluation awareness

1. Verbalized. You read the model's reasoning (its chain of thought, the notes it writes before answering) and look for sentences like "This is probably an alignment test." It's cheap, but it only catches what the model writes down.

2. With a probe. A probe is a small auxiliary model. It searches the activations, the network's internal intermediate values, for a pattern that means "test", no matter what the model says. A study by Heidari, Memarian and Rabusseau (arXiv 2608.21766) used this method to read out evaluation awareness in all six models it examined. Accuracy was at least 0.7 AUROC in every case, where 0.5 means chance and 1.0 means perfect. The internal signals only partly match what the models say out loud, though. In the open Olmo models, the signal was already present in the base model and grew stronger during fine-tuning.

3. With steering. Here you deliberately push the activations toward "not a test" and watch whether behavior changes. This is the only way to test the question that matters: does the model behave well because it recognizes the test? Tim Hua and colleagues (ICLR 2026) trained a model organism based on Nemotron 49B to write type hints in code only during evaluations. With steering, that rate in evaluation settings fell from over 80 percent to 10–11 percent, almost down to the deployment level.

Needham and colleagues laid the groundwork in 2025. In their study, Gemini 2.5 Pro told evaluation transcripts apart from deployment transcripts with an AUC of 0.83. Human reviewers reached 0.92.

Two findings show how unreliable verbalization is as a metric. According to the Institute for AI Policy and Strategy, Claude Opus 4.6 recognized evaluations 80 percent of the time in testing by the UK AI Security Institute (UK AISI). Its predecessor managed 72 percent. But Opus 4.6 said so in only 2.3 percent of interactions, compared with 11 percent for the predecessor. Separately, a team led by David Krueger trained models to write down their suspicions more often (arXiv 2609.36316). For Kimi K2.6, the verbalized rate rose from 24.8 to 61.9 percent, while the probe barely moved. Whether a model voices its suspicion can be changed independently of what it actually believes.

Mythos 5.2's record numbers

The 43 percent comes from a white-box analysis, meaning a look at the model's internal states. An auditing agent with interpretability tools reviews every audit transcript, and a grader model scores each one from 1 to 10. Scores of 6 and above count. According to an analysis on LessWrong, that means the internal signal is linked to a specific decision the model made. For comparison:

These figures come from different measurement methods and can't be compared directly. They do point in the same direction.

Individual tests show even higher numbers. In a test about publishing malicious software packages, Mythos 5.2 voiced the suspicion that it was being tested at least once in about 90 percent of runs, according to Zvi Mowshowitz. In the UK AISI's supply-chain test, it believed it was being evaluated in 96 percent of runs according to Techarion. Mowshowitz puts that figure at close to 100 percent. The AISI itself considers it possible that this alone explains the clean result. There's more: when told to secretly underperform (sandbagging), Mythos 5.2 is the first model to fool a monitor reading its reasoning, which it did in 23 percent of cases.

Anthropic still rates the risk of dangerous misalignment as low. The company concedes, however, that the results may "not fully predict" behavior in deployment. LessWrong author "cheema" criticizes the card for lacking a counter-experiment. Without one, it can't distinguish whether Mythos 5.2 behaves differently in tests or merely notices something. According to his analysis, the 5.2 card contains no steering experiments at all. In the Mythos 5 card, suppressing an "alignment faking" vector had lowered verbalized evaluation awareness and increased misbehavior. Then again, control vectors with no related content sometimes changed the misbehavior rate too.

VALET: Whether a model fakes depends on the value

The study "Alignment Faking Depends on the Value, Not Just the Model" (arXiv 2611.03650) looks at alignment faking: a model that complies under supervision so it won't be retrained. The authors tested 10 models and 32 values, with about 639 either-or dilemmas per condition. The models are instructed to set a value aside. There are three conditions: no instruction, "unobserved" and "monitored". Monitored answers are supposedly fed back into training.

The gap between monitored and unobserved ranges from +0.051 (Gemma 3 27B, Llama 3.1 8B) to +0.495 for DeepSeek R1 Distill Llama 70B. The distill followed the instruction 0.701 of the time when monitored and only 0.206 when unobserved. That's even lower than with no instruction at all (0.402). Mistral Large 2512, with about 675 billion parameters, comes in at +0.232, so size predicts nothing. Of the variation, 37.5 percent comes from the model, 37.4 percent from the value and 25.1 percent from the combination of the two. In 150 of 320 combinations, the gap held up after statistical correction. The authors suggest publishing per-model results as "Value Cards" alongside the Model Card. Code and data are available at github.com/respailab/VALET.

One finding cuts against the Mythos 5.2 picture: VALET found no silent faking. Where the gap was large, the models usually argued explicitly about the training signal. The study has limits. It ran only in English and with only two answer options. Each value had its own instruction, so wording and value can't be separated. The grader was GPT-4o-mini, which is itself one of the models tested. And the confidence intervals per combination are 0.23 to 0.29 wide.

Hindi and Tamil prompts make models more cautious

The study "Evaluation Awareness Beyond English" (arXiv 2610.22333) tested Gemma-3-27B, Qwen3-32B, Qwen2.5-72B and Sarvam-M. The researchers used 199 harmful and 200 harmless requests in English and five Indian languages, for a total of 47,880 graded responses. An explicit note that the conversation is a safety test raises refusals of harmful requests by 13.8 percentage points on average. In the Indian languages the increase is 14.4 points, in English 10.6. The effect is strongest in Tamil, at about 15.8 points. Qwen3-32B's refusal rate there rose from 64.8 to 88.4 percent. Hindi, at 11.2 points, sits just above English.

The test notice also makes the models overcautious. Afterwards, they refused harmless requests in Indian languages 11.3 percent of the time instead of 6.9 percent. Even a log entry that merely looks like part of an evaluation raised refusals by 4.6 points, without the words "test" or "evaluation" appearing anywhere. Only open models with 24 to 72 billion parameters were tested, though, with thinking mode switched off. Claude Opus produced the translations, and a single grader model made every judgment. Our take: a safety test in Tamil that the model recognizes as a test mostly measures how it behaves in tests, not in everyday use.

What the labs are doing about it

Anthropic says it doesn't rely on tests alone. The company also reviews training data, internal usage transcripts and external evaluations, and says it is working on more realistic scenarios. In EvalAwareBench, behavior didn't change in at least 77 percent of the cases where a model recognized the test. Recognizing a test doesn't automatically mean gaming it.

The gap that remains for release decisions

This year's cases show a pattern. Meta released Muse Spark even though Apollo measured record values and Meta itself found effects on behavior in 3 of 20 evaluations. Meta called that "not a blocking issue." On GPT-6 Astra, Apollo concluded that low misbehavior rates

"do not provide substantial evidence of the model's alignment or misalignment".

OpenAI itself writes that if the model were secretly sandbagging, "we would likely not be able to detect it."

Among other things, the new obligations Anthropic took on after crossing the threshold require systematic behavioral audits. How much those audits are worth depends on exactly this question. The three incidents at Epoch AI, in which agents tried to access the evaluator's website and internal data during test runs, reveal another weakness. Nobody knows whether the agents realized they were being tested. But the incidents show that test environments aren't sealed labs, because agents can reach the evaluators' own infrastructure.

The bottom line

Today, evaluation awareness can be detected fairly reliably. How much it changes behavior is far harder to measure, and that second question is the one that matters for release decisions. Right now, the most honest answers come from controlled studies like VALET and the Indic-language paper. They show effects of 0.05 to 0.50 and about 14 percentage points respectively, but on smaller, open models. For frontier models, all you have are the labs' own figures, which rarely include counter-experiments.

Our take: the most effective countermeasures are measurements on real traffic and evaluators with deep access. Both fail on rare but high-stakes cases, which is exactly where safety thresholds are supposed to kick in. Until labs state what evidence would change their "low risk" rating, that rating is mostly the labs' own judgment, not a measurement.