Cloudflare blocks AI crawlers by default in a major win for web publishers

Cloudflare has rolled out a major policy update where new sites will now block AI crawlers by default. This is a critical turning point for content creators and publishers, who have long-faced bots scraping content without compensation or traffic returns.

Historical context: How it used to work

Traditionally, the web operated on a “content-for-traffic” model: search engines crawled sites and sent visitors back via links.