One line in robots.txt: OpenAI, Google, Microsoft and Anthropic back a shared AI opt-out

What this is about

If you run a website and don't want your text, images or videos used to train AI models, you've so far had to maintain separate rules for each provider. That could change: OpenAI, Google, Microsoft and Anthropic have announced they will support a shared signal in robots.txt. The robots.txt is the text file in a website's root directory where owners tell search engines and other crawlers (automated programs that fetch web pages) what they may access.

According to The Next Web, the proposal comes from a draft by Kevin Kelley of Anthropic. He submitted it to the Internet Engineering Task Force (IETF) on Friday (October 30) and presented it on the mailing list of the AI preferences working group (AIPREF). On the same day, Google, OpenAI and Microsoft said they intend to support the rule. The Korean outlet ZDNet Korea reported on it on November 1.

How the shared signal works

Until now, site owners had to address individual crawlers or policy identifiers, such as OpenAI's "GPTBot" or Google's "Google-Extended." The new draft introduces a shared "policy token." That's an identifier that doesn't refer to a specific crawler but to an intended use. It's called __usage_ai-training.

According to ZDNet Korea, two lines are enough for a full opt-out:

User-Agent: __usage_ai-training
Disallow: /

This means none of the content should be used to train generative AI models. As usual with robots.txt, you can exempt individual paths or explicitly allow them. According to The Next Web, the example in the draft blocks an entire site except for the press kit.

The providers plan to apply the signal alongside their existing settings. Kelley wrote that it complements Anthropic's existing crawler controls rather than replacing them. He also cautioned: "It will take some time for us to add support in our services and to work through the inevitable complications and edge cases."

Who joins, and when

Microsoft is making the biggest changes. Since 2023, the company has used the meta tags "NoArchive" and "NoCache" to keep content out of AI answers and AI training. According to ZDNet Korea, sites that use them must switch to the new method by March 1, 2027. Microsoft is also introducing its own page-level setting for "grounding." That's the term for when an AI answer draws live on a current web page. One option allows Microsoft's own products such as Copilot, but blocks the grounding services Microsoft sells to other companies.

Krishna Madhavan of Microsoft Bing wrote: "Publishers shouldn't have to choose between being discoverable in Bing and deciding how generative models use their work." Microsoft also warns that blocking AI grounding will "significantly reduce" the visibility of your content in AI-generated answers. Your ranking in Bing search results won't change as a result.

Google Search Advocate John Mueller pointed out, according to The Next Web, that sites that allow Google-Extended can appear in Gemini app answers and receive clicks.

What the signal doesn't do

The limits are significant:

• No access block: According to the draft, the token is a preference about how content should be used, not a command about crawling. Kelley said on the mailing list that it's about usage, not crawl traffic, and that the two are "deliberately separate."
• No effect on search: Visibility in search results is untouched. According to ZDNet Korea, AI answers that draw on search results aren't directly covered either.
• No entry means no preference: A site without a group for the new token has expressed no preference. Blanket rules such as User-Agent: * don't count, as a Japanese blogger (note.com) also stresses.
• Private contracts take precedence: Agreements between a site and an AI company can override the signal.
• No payment: Microsoft makes clear that the settings create no licensing or payment arrangements.
• Not a standard: The draft is an individual submission. According to the IETF Datatracker, the IETF hasn't endorsed it and it has no formal status. Anyone who ignores it has nothing to fear.

Background and reactions

Google sees the proposal as consistent with the EU Code of Practice for general-purpose AI models (AI Act), which recognizes robots.txt as an important way to express machine-readable preferences. The draft thanks contributors from the Financial Times, the BBC, Bertelsmann and the European Publishers Council, among others.

Meta communications chief Andy Stone spoke out on X in favor of a shared signal, according to The Verge (cited by TNW), but left open whether Meta would honor it.

The topic is contentious. In July, Cloudflare told AI crawlers to pay or be blocked. There's also a dispute over platforms that use user content for AI by default. According to Yahoo/IGN, Amazon was sued in the US in August over a Twitch setting that automatically uses streamers' content for Amazon's generative AI. Twitch product chief Mike Minton said of it: "If it were opt-in, nobody would agree." In the UK, the competition authority CMA is also examining Meta's terms of service, according to note.com.

Our take

Standardization makes practical sense. Instead of constantly adding new crawler names, there's one line for everyone. That lowers the barrier, especially for small site owners. That all four big providers are on board is a strong signal.

There are real weaknesses, though. Almost everything starts only in 2027, the proposal is non-binding, and there's no way to check whether providers comply. Key players such as Meta haven't committed. The signal also only protects against future training, not against past scraping, and it can cost you visibility in AI answers. Whether this becomes a reliable standard will only be clear once the implementations go live.