Cloudflare Sets September Deadline for AI Companies to Separate Crawler Traffic
Cloudflare has announced a new policy that forces AI companies to clearly separate their web crawlers based on their purpose. Under the new rules, AI companies must differentiate between crawlers used for traditional search indexing and those deployed for AI training or agent operations.
The deadline for compliance is September