91%, 12%, four Uber rides: why AI agent benchmarks score so differently

What's going on

On November 2 and 3, 2026, three major benchmarks for AI agents came out. Agents are systems that don't just answer but click, edit files and finish tasks on their own. At first glance, the benchmarks contradict each other. OpenAI's LifeVal gives GPT-6 Astra 80.6% on everyday tasks. On Physical-Engineering's Last Exam (PLE) from Google and Mercor, the best system meets 91% of all individual requirements but still stays below 50% overall. On GDPx from research firm SemiAnalysis, the leader, Claude Opus 5.5, scores 12%, and under strict grading it scores practically zero in finance and law. Then on November 4, Uber reported that in its busiest October week, a grand total of four rides were booked through AI agents.

Our thesis: there's no contradiction. The numbers measure different things, and above all, they're calculated differently. Once you know how the math works, you can read the numbers.

Three tests, three ways to score

OpenAI derived LifeVal from two time-use studies and a survey of 3,995 people. Around 100 users wrote the tasks along with grading rubrics, meaning checklists used to judge a solution. The metric is called "Average Reward": attempts are averaged per task, then all tasks are weighted equally. According to secondary analyses by Apidog and Japanese author kazu on note.com, there are 169 tasks. If you meet half the rubric, you get half the points. So 80.6% does not mean four out of five errands succeed. The score was also measured at the highest compute setting, "xhigh," and a chart note ("No WTP weighting") points, according to Apidog, to an unpublished variant weighted by willingness to pay. The widely quoted 100% scores on individual tasks, such as ordering halal groceries on a budget, are "best results," not averages, according to dev.to. The trend is also striking: from GPT-5.2 (49.3%) to GPT-5.4 (73.0%), the score jumped 23.7 points, but from GPT-5.6 (80.0%) to GPT-6 Astra it rose just 0.6.

PLE puts a number on the gap between "almost right" and "usable." For tasks created with an agent pipeline, the paper gives the formula s = 0.4·p + 0.6·G·O′. Here, p is the share of requirements met. G is a gate: it's 1 if every hard safety and feasibility check passes, and 0 otherwise. O′ measures how close the cost is to a proven lower bound. Do the math: a design that meets 91% of requirements but fails one hard check gets at most 0.4 × 0.91 = 36.4%. That's exactly what happened with Opus 5.5 in Claude Code: on average it met 91% of requirements, but only 41.3% of its designs passed every hard check. Chinese tech site Leiphone sums it up as "91% ≈ 0%" and gives an example: a cable tray crosses the sprinkler line, and dozens of pages of correct calculations become worthless. The problem, Leiphone says, is coupled requirements, not computing power. Fix one requirement and you may break the next.

GDPx is the strictest. According to SemiAnalysis, the metric is "pass@1 over three attempts": each attempt counts as pass or fail, then the results are averaged. All eleven domains count equally, no matter how many tasks they contain. That shifts the weights. Chip design makes up 34 of the 206 tasks (16.5%) but counts for only one eleventh (9.1%). In finance and law, nearly every model scored zero under the binary grading that was originally planned, so SemiAnalysis reports partial credit there. By our math, that means 2/11 (18.2%) of the overall score rests on a different yardstick than the rest. On top of that, safety filters drag down Opus 5.5's and GPT-6 Astra's scores in cybersecurity, biology and alignment, on tasks the models sometimes refuse.

One run or every time: pass@k and pass^k

How often a test is run is the second big lever. pass@k counts a task as solved if at least one of k attempts succeeds. That's interesting for research but too optimistic for real-world use. pass^k requires all k attempts to succeed. The company Sierra introduced it with its τ-bench to measure reliability. There, GPT-4o dropped to about 25% at pass^8 in the retail scenario, roughly 60% below its single-attempt score.

A simplified example, assuming independent attempts: an agent that solves a task 80% of the time scores 99.2% at pass@3 and 51.2% at pass^3. At a 50% single-attempt success rate, it's 87.5% versus 12.5%. From the same model, you can spin headlines anywhere from "near perfect" to "useless."

PLE and GDPx both report averages over three runs, not best scores. PLE shows how much the runs vary: per task and harness, the best and worst of three runs differ by 15.4 points on average. 105 of 595 combinations swing by 30 points or more. Across the whole set, these outliers average out. For your specific task, they don't. For LifeVal, as far as we know, OpenAI doesn't state the number of attempts.

Who's measuring, and who profits?

Self-grading: LifeVal is a test by OpenAI, with OpenAI rubrics, of OpenAI models. Tasks and graders aren't public, and competing models are missing. That makes a comparison with Gemini or Claude impossible. OpenAI's predecessor GDPval drew the same criticism: its automated grader agreed with human experts only 66% of the time, while two humans agreed with each other 71% of the time.

LLM as judge: PLE isn't free of this either. Tasks written by experts are graded by an LLM judge, meaning a language model acting as examiner, and that judge is none other than Claude Opus 5.5, the winning model. The authors looked into it. With Claude Opus 5 or Gemini 4 Argon as judge, 95.8% and 95.5% of verdicts matched respectively, and the rankings stayed largely stable. They estimate the self-preference effect at 2.3 points. That's honest disclosure, and it still matters when margins are tight.

Commercial interests: GDPx is mostly private: there's one public task per domain, and the rest come from 14 data partners. Nobody has independently verified it so far. In its write-up, SemiAnalysis invites labs to have their next models evaluated and says it wants to "showcase the best data providers." SemiAnalysis doesn't disclose terms. That's why the dev.to author calls GDPx a paid product. Google, for its part, is both publisher of PLE and the vendor of the Gemini models, and runs 13 models it sponsors for free on its own servers. That said, an Anthropic model won.

The side finding: the model matters, the harness costs

A harness is the working environment around the model: tools, file access and the loop in which the agent plans and acts. Well-known examples include Claude Code, Codex, OpenHands and Pi. PLE compared 14 models across seven harnesses. Models differ by up to 50 points, while harnesses differ by a median of just 3.0 for the same model. Opus 5.5 scores between 47.3% and 50.6% across all seven (all 85 tasks). There are outliers: Gemini 4 Argon drops to 31.1% in OpenHands but reaches 43.2% in Claude Code.

Costs, on the other hand, vary a lot. For Opus 5.5 per task:

According to the paper, GPT-6 Astra in Codex reaches 48.3% for $9.15. Careful when quoting: that 48.3% applies to all 85 tasks. Opus 5.5's identical 48.3% on the leaderboard refers to the 56 hidden test tasks, where GPT-6 Astra scores 45.8%. Our take: when choosing an agent, pick the model first, then the harness based on price and usability. Choosing a harness is more a cost question than a quality question.

And the four Uber rides?

On the earnings call, Uber CFO Balaji Krishnamurthy said the company handled around 320 million rides and orders during Halloween week:

"On Uber Eats, it was about 500 orders through AI agents. On rides, it was 4 in total."

That's about 1.6 agent bookings per million transactions. CEO Dara Khosrowshahi called the response "very, very early and very limited" and offered the explanation himself: booking a ride in the app takes 50 seconds on average, reordering your favorite meal 90 seconds. A week earlier, DoorDash also reported only "a few hundred" agent orders. This number doesn't measure what agents can do. It measures whether people use them at all. An agent can do well in a test and still be pointless in daily life if the direct route is faster.

Five questions for any agent percentage

1. What counts as success? Partial credit per requirement, all-or-nothing, or a gate that zeroes everything out on a single safety failure?
2. How many runs? One run, an average, the best attempt (pass@k), or every time (pass^k)? Are best scores being sold as headlines?
3. How is it weighted? Do tasks or domains count equally, and does the yardstick change between domains?
4. Who built it, who grades it, who profits? Is the vendor grading itself? Is a language model doing the grading, maybe the one being tested? Are the tasks public and verifiable?
5. What exactly ran, and at what price? Which compute setting, which harness, how many dollars and hours per task? Does the task resemble what you actually want to delegate?

The bigger picture

This week's three benchmarks aren't as far apart as the headlines suggest. They answer different questions. LifeVal asks how much of an everyday errand gets usefully done. PLE asks whether an engineering design is fully safe and buildable. GDPx asks whether a hard expert task is passed in full. So the models can already handle a lot of the individual steps, but rarely deliver a completely error-free result. On PLE, even the leader scores below 50% on 48 of 85 tasks.

In our view, PLE is the most transparent, with a public portion, a disclosed formula and its own check of the judge model. LifeVal is the least verifiable. GDPx is demanding but largely a black box with mixed yardsticks. The open question is whether people need agents in daily life when the app is faster. For now, the four Uber rides are the most honest data point on that.