Major AI companies back a shared robots.txt opt-out signal
Google, OpenAI, Microsoft, and Anthropic are backing a proposed common signal for robots.txt that publishers could use to indicate whether their content may be scraped for AI training.
The proposal, put forward by Anthropic technical lead Kevin Kelley as part of Internet Engineering Task Force work, would add a