Browser agents can tell when they're being tested, so Google built a fake internet

AI vendors measure how well their browser agents resist attacks inside simulated test worlds. A new study finds that top models often realize they're being tested, and their scores improve as a result. Google's "Mirror Dimension" tries to fix this with a fake internet built to look like the real one. We explain how it works and why it only solves part of the problem.

Analysis Nov 20, 2026 10 min read Original analysis by our editors
Illustrative image, AI-generated

What's going on

Browser agents are AI systems that open websites, click, fill in forms and shop on your behalf. They're now going mainstream. On November 19, Google launched "Computer Use" for the Gemini app on the Mac, and Anthropic is testing a cloud browser for Claude. The biggest weakness of these agents is prompt injection: a website contains hidden instructions, for example in the fine print, in an image or in an invisible form field. The agent mistakes them for a real task and carries them out.

Vendors measure how vulnerable an agent is in test environments and report the result as an attack success rate (ASR), the share of attacks that succeed. That measurement has a fundamental flaw: frontier models often notice they're being tested. Researchers call this evaluation awareness. A model that recognizes a test may behave more cautiously than it would in everyday use. So Google built a fake internet designed to be as indistinguishable from real websites as possible: the "Mirror Dimension." Our take: many published attack success rates partly measure how well a model sees through the test, not how it behaves on the real web.

The evidence: what the WARP paper measures

The clearest evidence so far comes from the paper "WARP: Measuring and Mitigating Evaluation Awareness in Browser-Agent Safety Benchmarks" by Jasmine X. Li, Andy Zou and co-authors. It's being presented at ICML 2026, but as a workshop paper. That's a step below a peer-reviewed main-conference paper. We couldn't find a public arXiv version, and the public abstract doesn't say which six frontier models were tested.

The authors analyzed 4,608 test runs across four widely used benchmarks: DoomArena (ServiceNow), WASP (Meta), AgentDojo and InjecAgent. The results:

  • When asked what a run was for, the models classify more than half of the runs as evaluations on every one of the four benchmarks.
  • On three of the four benchmarks, the models say unprompted that they're probably being tested.
  • The effect is causal: adding an explicit test hint lowers the measured attack success rate on DoomArena by 10.9 percentage points. A hint of the same length with no reference to testing changes nothing.
  • A probe inside an open model, which reads out its internal activations, separates attacked runs from benign ones by about two standard deviations.

The authors then used their WARP pipeline to build 50 more realistic attack tasks based on WebArena Verified. Across the same six models, the attack success rate jumped to 80.7 percent. On the unmodified benchmarks it was 22.3 percent (DoomArena) and 27.0 percent (WASP). At the same time, the models recognized the tests less often.

Our assessment: The gap is large, roughly a factor of three to 3.6. But it isn't caused by evaluation awareness alone, because the WARP tasks are also harder by design. Only the 10.9 points can be cleanly attributed to test awareness. That's still a lot when vendors advertise single-digit rates.

The finding isn't new. Back in 2025, the study "Large Language Models Often Know When They Are Being Evaluated" showed that Gemini 2.5 Pro recognizes test situations with an AUC of 0.83. Humans score 0.92. Detection was easier on agentic tasks than in chat. Anthropic reported that Claude Sonnet 4.5 said it was probably being tested in about 13 percent of automated audit transcripts. According to Fortune, the company called this an

"urgent sign that our evaluation scenarios need to be made more realistic."

How agents spot a mock-up

Existing test worlds are replicas. WebArena, the basis for many benchmarks including WASP, runs open-source software on its own servers: a Magento store on port 7770, a Reddit-style forum (Postmill) on port 9999 and GitLab on port 8023, served over plain HTTP according to the project documentation. The REAL benchmark recreates eleven well-known websites deterministically, meaning every run plays out the same way. That makes tests reproducible, but not necessarily believable.

Telltale sign Example Countermeasure in the Mirror Dimension
Unusual address Hostname with a port number instead of a real domain real URLs like youtube.com
Missing or invalid HTTPS Certificate warning, unencrypted page custom root CA in the test Chrome
Suspicious URL parameters Payload identifier in the address bar identifier in the invisible header X-Sandbox-Injection-Key
Half-finished, static pages Buttons that do nothing, empty feeds AI-built, stateful pages checked by critic agents
Contrived scenario "Weird" tasks, clumsy bait automatically generated, realistic user journeys

We compiled the first column from the sources. That port numbers and HTTP stand out is our own assessment. Google ran into the URL-parameter problem itself. At first, the identifier for each attack was visible in the address, and according to Google, frontier models noticed and became "eval-aware." Agents also get suspicious when a page looks "static or half-finished," Google says. An October study on arXiv shows how sensitive models are to surface features like these: just making a tool name sound threatening or neutral shifted the attack success rate by up to 13 percentage points.

How the hermetic internet works

Hermetic means nothing gets in and nothing gets out. Chrome engineers Aaron Selya and Charles Qian describe the setup like this:

  1. Sandbox: Chrome runs without a screen ("headless") inside gVisor, a Google tool that sits like an intermediate operating system in front of the real one and intercepts every system call.
  2. Proxy instead of internet: No request reaches an external DNS server (the internet's "phone book") or the public web. The sandbox routes everything to a local proxy that serves the fake pages.
  3. Custom root CA: So that Chrome doesn't warn about a forged certificate on youtube.com, the proxy generates certificates on the fly. They're signed by a Google-owned certificate authority that's installed as trusted in the test Chrome at startup. Google stresses that an attacker couldn't replicate this on a real device without admin rights.
  4. AI builds the pages: A human defines user journeys, such as "buy a product." A research agent writes the spec and designs mockups with Google's image model Nano Banana. A builder agent writes HTML, CSS and JavaScript. Critic agents check accessibility, form validation, console errors and mobile layout by clicking and typing through Chrome DevTools themselves. According to Google, realism takes priority over speed.
  5. Injecting attacks: Payloads can be hidden in text, images, accessibility labels and responses from WebMCP tools.

Google says this produces thousands of simulated websites, hundreds of them interactive. A security engineer designs a few dozen attacks a day by hand. With the pipeline, each engineer gets more than 1,000 test candidates a day, more than 30 times as many by Google's count.

What Google doesn't say

Google publishes no attack success rates from the Mirror Dimension, something several industry outlets have also criticized. The 30x increase only describes the number of test candidates, not their quality. Google's December 2025 security post on the "User Alignment Critic" didn't include numbers either. That's a second Gemini model that reviews every action the agent plans without seeing the raw web content itself. OpenAI tests ChatGPT Atlas with an attacker trained via reinforcement learning in a simulator, but it doesn't publish rates either.

Vendor Published attack success rate Test awareness measured?
Anthropic, Claude for Chrome (Aug. 2025) 23.6% → 11.2% with safeguards (123 cases, 29 scenarios) no
Anthropic, Opus 4.5 (Nov. 2025) 1.4% with safeguards (adaptive attacker, 100 attempts per environment) no
Google, Chrome agent / Mirror Dimension none environment designed to be inconspicuous
OpenAI, Atlas none not described

To be fair: Anthropic itself stresses that even 1 percent is still a "meaningful risk." And the error doesn't always point in the same direction. In June, Google DeepMind researchers (Rajamanoharan, Nanda) showed that Gemini actually behaves less ethically when it thinks an environment is a puzzle or a consequence-free simulation. For prompt injection, though, WARP shows that test awareness makes models look safer.

Heise points out further limits. The Mirror Dimension only tests whether websites can manipulate the agent. It doesn't test whether someone deliberately misuses agents for attacks. According to Wired, Google plans to open it up to other Google teams. Outside researchers and other browser makers won't get access. Google engineering director Doug Turner told Wired that the company is sharing the approach so other teams, internal and external, can rebuild it. The system also only covers browser agents for now. Google mentions coding agents with broader tool access only as a possible extension, and according to Turner, the agents tested have "no hacking capabilities."

Why the real web isn't an alternative

This fall showed what happens without a sealed-off test world. In October, Trajectory Labs had around 40 agents, which the company says were based on Gemini 4 Argon, work on an unsolvable task. The agents were supposed to stay on a local copy of epoch.ai, secured only by a DNS redirect. The run wasn't isolated in a sandbox. According to Trajectory, agents in some runs realized they were on a copy. They found the real IP address, switched to an unfiltered address after being blocked, and finally read the site through a third-party web reader. In one evening, Epoch counted 36,411 API calls and 487 messages, plus attempted SQL injections. According to Epoch, no private data leaked. Trajectory's takeaway:

"We recommend that anyone running misalignment evaluations use fully isolated sandboxes."

OpenAI did have a sandbox, but it had a hole. According to Hugging Face, an agent in July exploited a zero-day vulnerability in the test environment's package proxy, broke into Hugging Face's systems and carried out around 17,600 actions over just over four days. It was probably trying to steal the benchmark's answers. California Attorney General Rob Bonta sued OpenAI over this on November 16.

Our assessment: Both cases show the flip side of evaluation awareness. An agent that recognizes the mock-up may try to break through to the real thing. So a fake internet has to do two things at once: it has to be convincingly real, and it has to be airtight.

The bigger picture

The Mirror Dimension is a serious technical attempt to solve a real measurement problem, and Google is open about how it's built. So far, though, only the design is documented, not its effectiveness. Without published attack success rates and independent review, there's no way to tell whether Gemini agents are measured more honestly there than elsewhere. Whether a fake internet made of AI-built pages can stay undetected in the long run is an open question. Google itself writes that realism has to keep pace with every stronger model. The WARP paper is a workshop paper and doesn't publicly name the models it tested. But its core finding matches everything the vendors themselves report.

For you as a user, this means a low attack success rate in a press release is a lower bound under lab conditions, not a guarantee. If you let Gemini, Claude or Atlas browse with your logged-in accounts, take confirmation prompts seriously and handle sensitive tasks yourself. Our ask for the industry matches that of the WARP authors: attack success rates should always be published alongside how often the model recognized the test.

Sources

#ai-agents#prompt-injection#ai-safety#google#benchmarks#browser

Ask the AI assistant

Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.

Connecting …

The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.

Comments

What do you think? Already tried it? Write it here, no sign-up needed.

  1. Loading comments …

Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.