Portal2Bench: Only GPT-6 Astra cracks the Portal 2 chamber that stumps every other model

What it's about

A hobbyist developer who goes by u/Deep-Garbage6832 on Reddit has built a test for AI agents called Portal2Bench. The project page is at kingstonyip.com/portal2bench. The idea: current language models have to solve the test chambers from Valve's puzzle game Portal 2 with no outside help. The models see nothing but screenshots, and they press keys virtually. An agent here means an AI model that plans and carries out several steps in a row on its own.

The result the developer highlights on Reddit: of the seven models tested, only OpenAI's GPT-6 Astra beat the chamber in Chapter 4, Level 1. According to his description, the model used cubes to line up laser beams until every relay lit up. Then it moved a portal so that a light bridge lay flat, crouch-jumped onto it and walked out. He says he solved the level differently on his own first playthrough: after many attempts, by throwing a cube at a turret.

The test is still running. According to the project page, many chambers haven't been played by every model yet because of limited compute and usage caps. Portal2Bench is not affiliated with or endorsed by Valve.

How the test works

Portal 2 is paused through the SAR plugin, an add-on from the speedrunning community. That way the model can think without the player character moving. The game only runs briefly when the AI sends a command (a tool call). The agent can:

• aim: turn the view and look in a given direction,
• move: W, A, S, D, crouch, jump,
• interact: the E key and the portal gun.

In return, it gets screenshots and its position in the level. It has its own folder for notes, but no command line, no internet and no access to the game console. The developer says the environment contains no walkthroughs, map names or cheats. The agent's file tools are limited to the notes folder, and a quality check flags any other access. If the character dies, it respawns at the last checkpoint, just like in the normal game, and the agent is only told that it died.

Each chamber is a fresh, independent run. It counts as solved once the agent reaches the next map within 300 plans (planning steps) and $30 in API costs. Costs are calculated at the providers' official list prices. The score covers 26 chambers from chapters 2 to 5. Each model runs in its maker's own agent environment (the "harness"): OpenAI's Codex, Anthropic's Claude Code or Google's Antigravity, always at the highest reasoning level ("max effort"). The only exception is Gemini 4 Argon, which runs on "high".

The results

Play time excludes the pauses for thinking.

OpenAI's models are clearly ahead. Sol, Astra and Terra each solve about three quarters of the chambers. Opus 5.5, the strongest Anthropic model tested, doesn't even get half. At a median of 74 cents per solved chamber, GPT-6.1 Terra is also by far the cheapest model. Leave Fable 5.1 aside and Astra needs the fewest plans. So far, Fable 5.1 has only been tested on 15 chambers. GPT-6 Luna also stands out: despite coming from OpenAI, it doesn't solve a single chamber.

The cost figures only cover solved chambers, so a model with few wins has only a handful of data points there. Be careful with the harness comparison too. Codex, Claude Code and Antigravity all work differently, so what's being measured is always the model plus its harness. The developer has promised a detailed write-up of the methodology, but it hasn't been published yet.

The inspiration: GPT-6 Astra beats the original Portal

Among the developer's named inspirations are cozyblaze, skipdot and the SIMA team, which works on game-playing AI agents at Google DeepMind. Cozyblaze made headlines shortly before: he had GPT-6 Astra play the original 2007 Portal all the way to the credits. Tom's Hardware, Tbreak and Gadgetreview were among the outlets that covered it.

The setup is similar. According to Tom's Hardware, the model controls the game through MCP (Model Context Protocol, a standard interface that lets AI models operate external programs) and a modified SourcePauseTool. Here's how cozyblaze explains it:

"The game stays paused while the model thinks. As soon as the model sends an input sequence, SPT unpauses and executes it."

The numbers, according to the reports:

• 23 hours and 43 minutes of total runtime, of which only about two hours were actual play time,
• 3,336 tool calls,
• $571.18 in API costs at list prices. Gadgetreview puts that at $10 per million input tokens and $50 per million output tokens.

Tokens are the small chunks of text and images a model processes. Every one of the thousands of decision steps adds a new screenshot, so the bill climbs fast. According to Tom's Hardware, cozyblaze didn't actually pay that amount: he used his $200-a-month Codex Pro subscription. Tbreak adds that Astra's new context management helped the agent remember things across the long chain of decisions. Tom's Hardware reports that cozyblaze has published the Portal agent's code on GitHub.

Cozyblaze is explicit that his run isn't a benchmark but a proof of concept. Still, he says, it feels like "a small glimpse of that original vision becoming reality". He's referring to OpenAI's 2016 goal of building a single agent that can beat lots of different games.

Criticism: solving or cribbing?

The biggest objection concerns training data. Gadgetreview and Tbreak point out that Portal has been documented in exhaustive detail for almost two decades, with walkthroughs, wikis and speedrun guides. A model of Astra's size has very likely seen that material during training. Tom's Hardware readers agree. One commenter calls it "practically unavoidable" that solution guides are in the training data.

So the run doesn't show that the AI figures out an unfamiliar puzzle on its own. What it shows is that a general-purpose model can analyze images, use coordinates and repeat commands until it reaches the goal. The same caveat applies to Portal2Bench: the environment contains no solutions, but the training data probably does.

The cost raises eyebrows too. Tbreak notes that $571 is more than many gaming PCs cost and far more than the game itself. According to Gadgetreview, The Verge dryly remarked that the run "probably only" took a small swimming pool's worth of water to cool the data centers. Gamers were lukewarm. The Portal2Bench developer writes that his project went down rather badly on r/Portal, since many players don't have much love for AI right now.

Our take

Game benchmarks like Portal2Bench are interesting because they test things classic benchmarks barely touch. A model has to understand a 3D space, figure out physics and game mechanics, plan over many steps and recover from mistakes, all from images alone. That's exactly what AI agents need when they're supposed to operate a computer. OpenAI explicitly markets GPT-6 Astra as a leader in computer use and multi-step tasks. Portal2Bench backs up that vendor claim, at least in this area: OpenAI's models are well ahead of Anthropic's Claude models here, and GPT-6.1 Terra is very cheap on top of that.

Still, caution is warranted. Portal2Bench is a one-person project. There's no methodology report yet, many runs are still pending, and there are no results from Google at all. With games this well known, it's almost impossible to tell whether the models really understand the puzzles or are recalling solutions from training. A test on a new, unreleased game would tell you a lot more. Even so, the trend is clear: general-purpose models are increasingly handling long, image-based tasks that used to require specialized, purpose-trained systems like DeepMind's AlphaGo. For now, though, it's slow and expensive.