Cloudflare Sets September Deadline for AI Companies to Separate Crawler Traffic
Cloudflare has announced a new policy that forces AI companies to clearly separate their web crawlers based on their purpose. Under the new rules, AI companies must differentiate between crawlers used for traditional search indexing and those deployed for AI training or agent operations.
The deadline for compliance is September 15. After this date, AI companies that fail to separate their crawler traffic risk being blocked by default on the many publisher sites that rely on Cloudflare's infrastructure.
The policy addresses growing tensions between AI companies and publishers over how web content is used. AI developers have increasingly scraped publisher content to train language models, often without compensation or clear permission. This move by Cloudflare gives publishers more control over how their content is accessed and used by AI systems.
Publishers using Cloudflare will be able to choose which types of crawlers they allow or block, giving them a practical tool to negotiate better terms with AI companies that want to access their content.