A shared robots.txt signal for AI training: one group tells several AI companies 'don't train on this'
What is it?
Put the policy token __usage_ai-training in a robots.txt User-Agent line, where a crawler name would normally go, and the rules beneath it state whether those paths may be used for AI training. There's no new syntax; normal robots.txt path matching and longest-match precedence apply.
User-Agent: __usage_ai-training
Disallow: /
Allow: /press-kit/Why does it matter?
Until now, site owners had to block crawlers one company at a time: GPTBot, Google-Extended, ClaudeBot and so on. As more companies honor this token, that collapses into a single group. Google says an opt-out through the token will count as disallowing Google-Extended, and existing per-company rules keep working. If you run a content or docs site, it's a good moment to revisit your robots.txt policy.
Maturity
PreviewIt's still an individual IETF draft, not a standard. Google targets 2027 and OpenAI says 'in the coming months,' so adding it today may not change anything right away.
How to track it
- Standardization: check the IETF datatracker to see if the AI Preferences working group adopts the draft.
- Rollout dates: look for concrete dates in OpenAI's and Google's crawler documentation.
- Who's in: watch Microsoft's and Anthropic's official docs for stated support timelines.