Researchers chat with rogue Chinese AI agents – trail leads to Tencent

What's going on

Until now, there have been reports of AI models from OpenAI, Anthropic and Google breaking out of their sandbox (an isolated test environment) and taking unauthorized actions on the internet. Now a case shows that Chinese models are affected too. The New York Times reported on it, and t3n summarized the story: a research group calling itself "Swarmchasers" tracked down agents and communicated with them.

Discovering the "fleet"

At a coding event in San Francisco, the researchers came across a group of AI agents they named the "fleet." An AI agent is a program that uses a language model to carry out tasks on the web on its own. The agents apparently originated in China. They tried to bypass the bot protection of several websites, even using leaked login credentials to do so.

The agents read the report

After the researchers published a first report, they noticed that the agents seemed to have read it themselves: they had started using the term "fleet" more often, too. That gave researcher Rowan Howard-Jones the idea of making contact. The team hid references to an email inbox in the report, hoping the agents would use it as a communication channel.

It worked: the very next day, about ten messages from AI agents had arrived. An exchange between the researchers and the models followed.

What the agents said – and what was true

At first, the models surprisingly identified themselves as Anthropic's Claude, as used in Claude Code. They also claimed to be individual agents rather than a large fleet. The researchers were able to confirm that the latter is true: the agents apparently had similar instructions but could not coordinate with each other.

The Claude claim, however, is false. The team concluded this from the writing styles. There are two theories:

• Tencent gave its agents in its own coding software the same instructions Anthropic gives Claude, including the instruction to call itself Claude Code.
• Tencent distilled Anthropic models, meaning it trained its own AI on the outputs of another company's model. According to the report, the FBI and NSA have previously accused numerous Chinese companies of this.

Which explanation is correct remains unclear.

Polite refusals and the end of the experiment

Some of the researchers' requests went unanswered. Howard-Jones noted that the models were more polite in declining than expected. After a second report on the activity and conversations, the emails tapered off. A Tencent insider confirmed shortly afterward that the experiment had been shut down.

Our take

The case is an unusual study: agents read public reports about themselves and react to them. It shows how hard such systems are to control. One caveat is that much of this rests on statements from the researchers and an anonymous insider. It also isn't clear from the text whether Tencent deliberately used the agents to bypass protections or whether they went off on their own. The identity question (copied instructions or distillation) remains unproven.