Introduction

The biggest names in artificial intelligence are rallying behind a new industry standard that gives publishers a simple way to control how their content is used for AI training. A newly proposed policy token could change how websites manage scraping across multiple AI platforms.

What Happened

OpenAI, Google, Microsoft, and Anthropic have announced support for a shared signal that lives inside a website's robots.txt file. The token lets publishers set their AI training preferences once and apply them across all participating companies, eliminating the need to configure settings separately for each crawler. The initiative comes from Kevin Kelley at Anthropic and is being standardized through the Internet Engineering Task Force. Major publishers have long struggled with the growing number of AI bots hitting their sites, and this standard aims to simplify that friction.

Why This Matters

Until now, website owners had to negotiate AI opt-out settings individually with each company, a process that was time-consuming and often inconsistent. With a single signal, publishers gain broader control while AI companies respect the preference across the board. The standard also signals a shift toward more transparent, cooperative relationships between content creators and the AI systems that train on their work. Meta has expressed support, though it hasn't committed to full adoption yet.

Key Takeaways

  • The new policy token lives in robots.txt and uses a standardized signal to communicate AI training preferences.
  • OpenAI, Google, Microsoft, and Anthropic are on board, with rollout timelines varying by company.
  • Microsoft is adding granular controls for opting out of both AI training and AI grounding.
  • The approach could reduce the flood of independent AI crawlers and give publishers more predictable control over their content's future use.

Conclusion

This new standard represents a significant step toward unified AI web governance. As more companies adopt the policy token, publishers should monitor their robots.txt files and evaluate how the signal affects their content's visibility and training use. The coming months will show whether the industry can move from fragmented opt-outs to a truly shared standard.