98% on the Factory Floor, 0% in the Benchmark: What Humanoid Robots Can Really Do

What this is about

This week brought three stories about humanoid robots that don't seem to fit together at first glance. Xiaomi released a video of humanoids working a full shift at its Beijing car plant. UBTech's U1, priced at around $25,000, was reported to be "launching in cafés and factories" in South Korea. And a research group from Singapore, Seattle, Stanford and Hugging Face announced results from its RoboColosseum benchmark: on the Unitree G1 humanoid, not one of five leading AI control models managed even a simple household task.

So can humanoids already do real work or not? The short answer: both numbers can be true, because they measure completely different things. The long answer is below, along with a checklist for the next success story that lands in your feed.

What the manufacturers report, and what's missing

Xiaomi has been releasing its numbers step by step. In March, a humanoid worked as an "intern" at a station that inserts self-tapping nuts. Over three hours it hit a 90.2 percent success rate and kept up with the 76-second takt time, the maximum time each step on the line is allowed to take. By July, the rate at the same station had climbed to 98 percent. According to Xiaomi, that's just one percentage point below trained workers (99 percent). Two new stations, sorting center-console side panels and folding boxes, each reached 90 percent at the time, according to TechNode.

On November 3, Xiaomi followed up with the eight-hour-shift video at the side-panel station. Chinese outlets such as The Paper have broken it down in detail. The work lasted 8 hours and 5 minutes; the video runs at 100x speed and is just under six minutes long. There was a charging break after 3:02 hours and another after 6:03 hours. At one point a part fell to the floor and was picked up by a wheeled robot. The footage shows two humanoids, a wheeled upper-body robot and an autonomous transport vehicle that brings in the boxes.

What matters is what's missing. Electrek and Génération-NT list it: no piece count, no success rate over the eight hours, no figure for how often humans intervened. Tech site SMZDM points out that Xiaomi itself is careful with its wording. The robot covers the daily workload of one station, which the company calls a "preliminary confirmation" that it's suitable for an entire line. Xiaomi CEO Lei Jun has named the problem himself: in the lab you're allowed lots of failures, but in the factory every move has to be right.

For a sense of scale, here's our own math: at a 76-second takt time, about 379 cycles fit into eight hours. At 98 percent success, that would mean seven or eight failed attempts per shift. Keep in mind, though, that the 98 percent applies to the nut station, not to the station in the shift video.

The UBTech story needs correcting. According to the manufacturer, the U1 is a lifelike companion robot with silicone skin and a bionic face, sold under the consumer brand UWorld. The Pro version costs 169,800 yuan, roughly $24,000. In South Korea, according to The Gadgeteer, it was bought by the Galaxy Robot Park amusement park in Seoul. The café trials with the Hollys chain are run by Beijing startup Anyverse Dynamics. The factory trial on an auto parts line (with Duksan JM Robotics and Ajin Industrial) is a UBTech project, but no report says which model is working there or what exactly it does. So the story tells you nothing about what the U1 can actually do on the job.

What the independent test measures

RoboColosseum (arXiv 2611.02497) comes from a group that includes the National University of Singapore, the University of Washington, Stanford, the Allen Institute for AI (Ai2) and Hugging Face. It tested five VLA models, short for vision-language-action models, which translate camera images and a spoken or written instruction directly into motor commands: GR00T N1.7 (Nvidia), π0.5 (Physical Intelligence), MolmoAct2 (Ai2), G0.5 and LingBot-VLA v2.

In the fine-tune track, every model got the same 100 teleoperated demonstrations per task (teleoperation: a human steers the robot while the system learns from it). Each was trained using its developer's official recipe, followed by 25 attempts per task. Two metrics were recorded. The success rate counts only fully completed tasks. The progress score measures what share of the sub-steps the robot managed.

Progress / success in percent, averaged over five tasks each. Source: RoboColosseum paper, Table 2.

The five G1 tasks took place in real homes in Malaysia, with a 150-second time limit: put a plate in the drying rack, carry an empty box, fold a towel, load a washing machine with a T-shirt and trousers, and hang a set of keys on a hook. That adds up to 625 attempts (5 models × 5 tasks × 25) and not a single success. According to the paper, the robot met even a quarter of the criteria in fewer than 40 percent of attempts. Some models got far with carrying the box and loading the washing machine but failed at the final steps. The towel and the keys were too hard for almost all of them.

Why both numbers can be true

Fixed station vs. unfamiliar home. Xiaomi's robot works at the same spot for months, with the same parts, the same lighting and the same box. RoboColosseum puts robots in environments the model has never seen. Yet generalizing to new situations is exactly what π0.5's developer advertises. Ai2's ToyBlocks evaluation system, released the same week, concludes that π0.5 and MolmoAct 2 generalize poorly despite extensive pretraining.

Lots of data vs. 100 demos. We don't know how much data Xiaomi collected for the station. According to SMZDM, the base model was trained on more than 100,000 hours of human motion data and about 10,000 hours of robot data. The benchmark provided 100 demonstrations per task.

Walking plus grasping. The G1 tasks require walking, crouching, turning around and passing an object from one hand to the other. None of that applies to two fixed arms, where π0.5 holds its 93.6 percent. Electrek notes that even in Xiaomi's video, wheels (the transport robot and the wheeled upper body) fill the gaps.

Success vs. progress. Manufacturer numbers are often defined more generously. Physical Intelligence reports a 94 percent success rate for π0.5 in unfamiliar test homes. But according to its own blog, that figure is averaged over sub-tasks, such as the share of items correctly put away. That's closer to RoboColosseum's progress score than to its strict success rate.

One counterexample shows that the G1 isn't fundamentally incapable. In the Zero-HLM paper, GR00T N1.7 reaches a 66.7 percent success rate on the G1 when trained on 4,800 simulation demos, compared with 46.7 percent using real-world data. But that involved three different tasks in a different setup, so the results aren't directly comparable.

Rankings don't carry over

Perhaps the most important finding concerns rankings. Kendall W measures how much several rankings agree, from 0 (not at all) to 1 (completely). Across the robots, the value is 0.24. A test winner on one robot tells you little about the next. GR00T N1.7 leads on the small SO-101 but falls far behind on two arms, with a 20 percent success rate.

In the second part, the open track, models ran without task-specific training in 5,699 blinded pairwise comparisons at 124 sites. Evaluators watched two runs side by side without knowing which models were involved. This ranking turns out to be a poor predictor of performance after training. The rank correlation ranges from −0.40 to 0.20 depending on the robot, so the order sometimes even flips. On the SO-101, even two halves of the sites agreed only 59 percent of the time. The rankings only stabilized at 90 percent after about 1,100 ratings.

General-purpose AI agents such as Claude Opus 5.5 (+12.1 progress points over the VLA models) and GPT-6 Astra (+11.0) actually outperformed the specialist models in the open track, while Grok 4.7 came in at −7.5. On hard tasks (difficulty 9–10), however, the full success rate of all systems dropped to 0.4 percent. The authors write:

"None of the systems reliably solves tasks in open-ended environments."

The measurement itself fluctuates, too. With ToyBlocks, Ai2 reports that the same model scored between 54 and 59 percent across four runs of the same 128 simulation episodes. Dieter Fox (Ai2, a co-author of RoboColosseum) told the Spokesman-Review about a robot that succeeded in one spot and failed in another: "For a human, that would be trivial."

Checklist: how to vet robot claims

1. What counts as "success"? The whole task, or an average across sub-steps?
2. How many attempts? With 25 attempts, a single failure is already 4 percentage points.
3. Fixed station or new environment? Weeks of practice at one spot are not the same as an unfamiliar home.
4. How much training for this exact task? No demo count means no yardstick.
5. Did anyone intervene? Teleoperation, helper robots, human resets. In 2024, Tesla showed off remote-controlled Optimus robots without disclosing it.
6. Time-lapse and edits? Charging breaks and glitches belong in the tally.
7. Who's measuring? The manufacturer itself, or blinded, independent evaluators?
8. Which robot? Results barely transfer to other hardware.
9. Throughput and cost? Parts per hour and cost per shift compared with a human worker.

Our take

To its credit, Xiaomi does publish takt times and success rates, which Electrek explicitly praises. But for the headline result, the eight-hour shift, the very numbers you'd need to check it are missing. RoboColosseum is the opposite: open data in LeRobot format on Hugging Face, evaluation code on GitHub, and failures published too. The authors themselves call it a first study. It covers only five tasks per robot, the training recipes come from the developers, and 100 demos may simply be too few for a humanoid with whole-body movement.

Our assessment: the numbers don't contradict each other. They show two ends of the same scale. Narrow tasks with lots of data at a fixed spot are coming within reach. The humanoid that puts away laundry in a stranger's home is not there yet. If you're buying or investing in humanoids, do as AI Weekly advises and demand tests on your own hardware in your own environment. A leaderboard from somewhere else won't cut it.