Major AI companies back a shared robots.txt opt-out signal
Google, OpenAI, Microsoft, and Anthropic are backing a proposed common signal for robots.txt that publishers could use to indicate whether their content may be scraped for AI training.
The proposal, put forward by Anthropic technical lead Kevin Kelley as part of Internet Engineering Task Force work, would add a “policy token” to the website directives already used to guide crawlers. Support from the four companies suggests that a single publisher preference could be recognized across multiple AI platforms rather than requiring separate settings for each provider.
The effort is intended to give publishers more consistent control over how their material is collected. Its broader adoption could also depend on how widely the signal is implemented and whether smaller AI companies and other crawlers follow the same approach.