miHoYo unveils "Gamer", an AI that plays Honkai: Star Rail with mouse and keyboard

The AI team behind miHoYo's Honkai series has shown off a model that plays video games the way a person does: it only watches the screen and controls the mouse and keyboard. Across 40 minigames it had never seen, it completed just under half of the tasks on average. The researchers stress that it isn't meant as a cheating tool.

Research Nov 17, 2026 6 min read Source: ITHome (CN)
Illustrative image, AI-generated

What's going on

miHoYo, the Chinese developer behind Genshin Impact and the Honkai series, has unveiled an AI model called "Gamer". It was built by the Honkai series' AI research team. According to Chinese tech site ITHome, the model can play 3D and 2D games on its own, the way a human does: it sees only what's on the monitor and operates the keyboard and mouse. The demo uses Honkai: Star Rail, one of the company's biggest hits. There's a research paper and a project page. So far, all results come from miHoYo itself. There are no independent tests yet.

Under the hood, Gamer is a vision-language-action model (VLA). That's an AI that understands images, processes language and turns both into actions. You mostly see models like this in robotics, where they use camera feeds to control a robotic arm. With Gamer, the image comes from the game and the "arms" are a mouse and keyboard.

How Gamer plays

The model doesn't need any view into the game's internals. It doesn't read memory or game state, and it doesn't use a game-specific API, the kind of programming interface that would let software talk to the game directly. Gamer sees only what a human player sees.

Gamer captures the game image five times per second (5 Hz). From that it generates control commands at a resolution equivalent to 30 Hz. These are real key states (pressed or not pressed) and mouse movements. So between two frames, the model plans several small inputs ahead. Think of a human player who doesn't make a fresh decision after every blink but carries out movements fluidly.

Because the approach relies only on the screen, mouse and keyboard, it can in principle carry over to lots of games. That sets it apart from classic game bots, which are usually built for a single game and tap into its internal data.

Training and tech

Long play sessions are hard for AI models. The longer a task runs, the more visual history the model has to keep in memory, and the slower it responds. Games, though, demand fast reactions. According to ITHome, the team combined several building blocks to tackle both problems, limited memory and latency:

  • Continued pretraining on more than 10,000 hours of recorded mouse and keyboard input from a range of games. The model learns which inputs people make in which situations.
  • Sparse Thinking Instruction Fine-Tuning. The model learns to "think" explicitly only now and then, instead of deliberating at length before every input. That saves compute time.
  • Time-aware reinforcement learning, meaning learning through rewards: the model tries things out and gets rewarded for success. "Time-aware" means the timing of its actions is factored in too.

On top of that comes a custom inference engine called "Streaming-vLLM". An inference engine is the software that runs a finished model. The name refers to vLLM, a widely used open-source program for this, adapted here for a continuous stream of frames. On an Nvidia H200, a data center GPU, the team says this brings the delay between request and response down from 1306 to 65.12 milliseconds. That's roughly a twentieth. Only at that speed does smooth real-time play become possible at all.

What the model can do

In demo videos from Honkai: Star Rail, Gamer completes main missions, explores the world and solves puzzles. The measurable numbers come from a different test: the variant trained with reinforcement learning, Gamer-RL, hit an average success rate of 49.7 percent on 40 minigame tasks it had never seen before. The team says that puts it ahead of general-purpose models that weren't specifically fine-tuned.

Metric Value
Image input 5 Hz
Output (keyboard/mouse) equivalent to 30 Hz
Training data more than 10,000 hours of mouse and keyboard recordings
Response time without / with Streaming-vLLM (H200) 1306 ms / 65.12 ms
Gamer-RL success rate (40 unseen minigames) 49.7%

The source doesn't give the model's size, the exact models it was compared against or what kind of minigames were used. And just under 50 percent also means roughly every second unfamiliar task still fails.

Cheating fears and what's being released

At first glance, an AI that plays games through the screen, mouse and keyboard is hard to tell apart from a human. That has raised concerns the technology could be abused for cheats and automated bots, for example to "farm" resources. The researchers therefore explicitly describe Gamer as an academic research project on general-purpose AI agents in interactive environments. They list possible uses:

  • Game testing: the AI plays through levels automatically and finds bugs.
  • Accessibility: helping people who struggle to use a mouse and keyboard.
  • Human-computer interaction research.

The team writes that it explicitly does not support using the model for cheats, unauthorized access, exploiting security holes or anything that harms users or undermines fair play.

Citing data security, miHoYo is initially releasing only general files from the early pretraining phase. There's no training data from Honkai: Star Rail, and there are no model weights trained or fine-tuned on data from the game. So you can't download the model from the Star Rail demos and run it yourself.

The bigger picture

Gamer is part of a growing research field: general-purpose agents that operate software the way people do, using only the screen, mouse and keyboard. Similar approaches exist in computer-use agents for office apps and in game agents from other labs. What stands out here is that a major game publisher is doing this research with its own game data, and that the team pushed latency down so far. 65 milliseconds is in the range of human reaction times, though on expensive data center hardware.

For miHoYo itself, automated testing is the obvious use. Big live-service games like Star Rail get new content all the time, and someone has to play through every new mission. An AI that handles that would save money.

Take the numbers with caution, though. Every figure comes from the developer. The 49.7 percent success rate shows the model still fails often on new tasks. And because the Star Rail weights stay under wraps, outsiders can hardly verify the demos. Despite the disclaimers, the risk of abuse remains a real issue: the better these agents get, the harder they are to catch with classic anti-cheat methods, which often look for tampering with game memory.

Sources

#mihoyo#ai agents#gaming#vla#reinforcement learning
Read the original source

Ask the AI assistant

Questions about this article, installation or background? Our assistant knows this page and runs on our own machine, not in someone else's cloud.

Connecting …

The Assistant is an AI (Gemma 4) and can make mistakes – please check important details against the source. Questions are used only to generate an answer and are deleted after one hour. Please don't enter any personal data.

Comments

What do you think? Already tried it? Write it here, no sign-up needed.

  1. Loading comments …

Comments are public. Links aren't allowed, and we delete abusive posts. You can delete your own comment at any time in this browser. Learn more in the Privacy Policy.