On 30 October 2026, Google, Microsoft and OpenAI announced they will honour a single shared robots.txt signal that lets a website opt out of AI training. Anthropic made no announcement of its own — but the signal’s author is an Anthropic engineer, and he said the same day that Anthropic had already started implementing it. Until now, a site owner who wanted out had to learn four different sets of rules and hope each company honoured them. Now one entry is meant to speak to all four.

How it works

Add this group to the robots.txt file at the root of your site:

User-Agent: __usage_ai-training
Disallow: /

That states a preference against generative AI training on the whole site. You can narrow it: disallow only part of the site, and carve exceptions back in with Allow: lines — the draft’s own example blocks everything except a /press-kit/ folder. The token must match exactly (apart from letter case). A plain User-Agent: * group does not count as an AI-training preference.

Note the important distinction: the token names a use, not a crawler. Ordinary robots.txt rules name a bot, such as GPTBot. This one names what the content may be used for.

What it does NOT do — read this before you add it

1. It does not stop crawling. The draft states plainly that the token “is not a directive for crawling behavior”. Bots may still visit and cache a page; your existing crawler rules still apply as before.
2. It does not remove what is already collected. It only affects future training. Content already in datasets or already trained into model weights is not undone.
3. It covers training only, not AI answers. It does not keep a page out of AI-written answers that quote live content, and it does not cover models used only for non-generative tasks. One exception to be careful about at Google: if you use the shared token without naming Google-Extended, Google plans to treat that as blocking Google-Extended too — and Google-Extended also covers Gemini grounding, the step that pulls live content into Gemini’s answers. So at Google this token can affect AI answers after all. If you want training out but Gemini citations kept, name Google-Extended explicitly in your robots.txt and allow it.
4. It is not yet a formal standard. The signal comes from an individual Internet-Draft by Anthropic’s Kevin Kelley, submitted to the IETF’s AI Preferences working group. The IETF has not endorsed it and it has no formal standing — it binds only the companies that choose to follow it, and the draft expires in May 2027 unless updated. Treat it as a voluntary, coordinated preference, not a legal or security control. Non-compliant scrapers can ignore it.

When each company starts honouring it

The timelines are staggered, so do not expect the same effect everywhere on day one:

• Anthropic — implementation already begun, according to the draft’s author (an Anthropic engineer) writing on the IETF mailing list; Anthropic itself published no separate announcement.
• OpenAI — aims to implement it “in the coming months”.
• Microsoft — will honour it from 1 February 2027, and will stop reading its old NoArchive and NoCache tags as AI settings after 1 March 2027. Microsoft is also adding a separate page-level setting for AI grounding (the step that pulls live content into answers such as Copilot), which is different from training.
• Google — aims to support it in 2027. Google’s John Mueller has said a site using the shared token without explicitly naming Google-Extended will be treated as blocking Google-Extended — which at Google also covers Gemini grounding, so name Google-Extended explicitly if you want training out but Gemini citations kept.

What this means for a Pakistani developer, designer or SEO

• If your content is your product (paid courses, client work, original tutorials), the shared token is now the cheapest way to state a training preference to all four labs at once — but it is a preference, not protection. Keep your existing crawler rules and do not assume it hides you from search.
• If you want traffic from search and AI answers, be careful: opting out of training is not the same as opting out of search, but Microsoft warns that blocking AI grounding “will significantly reduce your content’s visibility in AI-generated experiences”. Decide separately what you want for training, for search, and for AI answers.
• If you run WordPress, edit robots.txt through your SEO plugin’s editor (for example Rank Math) and then check the public robots.txt file actually serves your new version — a server-level file can override the plugin’s version, which is exactly the kind of silent override worth checking before you assume a change has taken effect.
• Re-check in early 2027. With Microsoft’s February 2027 start and Google’s 2027 rollout, the practical effect of this token will change over the next few months. Set a reminder to re-test rather than assuming today’s behaviour holds.

Bottom line

One line now replaces four separate opt-outs for AI training — a genuine simplification. Just do not mistake it for a block on crawling, a removal of past data, or a guarantee. It is a stated preference, honoured on different dates by different companies, and it works best alongside — not instead of — your existing robots.txt rules.