The web's traffic cop just changed the default
Cloudflare, which sits in front of a vast share of the world's websites, has issued the AI industry an ultimatum with a date attached. Starting September 15, 2026, its default settings will block mixed-use crawlers - bots that blend traditional search indexing with AI training and agent traffic - from any pages that host ads, unless the site owner deliberately opens the gate. The new defaults apply to new Cloudflare customers, new sites from existing customers, and all existing free customers, which in practice covers an enormous slice of the open web. The logic, per the company: most site owners want to be discoverable via search and even AI services, but they want protection against their intellectual property being harvested for free as the price of that discoverability.
The Google subtext, said out loud
Cloudflare's announcement barely disguises its target, noting that the world's largest search engine enjoys about twice the information access of other AI companies because remaining discoverable in its search index has been hard to separate from feeding its AI. Google has pushed back on that framing before, pointing to its Google-Extended control that lets sites opt out of training and AI products without losing Search placement - though its flagship Googlebot still crawls for Search features including AI Overviews and AI Mode, which is precisely the bundling Cloudflare wants unbundled. CEO Matthew Prince anchored the urgency in a startling milestone: the majority of internet traffic is now non-human, a threshold crossed roughly a year ahead of expectations.
From Pay Per Crawl to Pay Per Use
The stick comes with a marketplace. Cloudflare's existing Pay Per Crawl scheme - which lets sites charge AI bots for scraping - is evolving into Pay Per Use, under which publishers get paid when their content creates value inside an AI product, not merely when it is fetched. Initial partners Ceramic.ai and You.com will pay publishers when content surfaces in AI search results or when premium content is accessed, with the model open for other AI companies to adapt. There is an efficiency dividend too: Cloudflare's data shows over half of AI crawl traffic is wasted re-fetching unchanged pages, so structured commercial access could cut publishers' bandwidth bills even as it opens a revenue line.
Two to-do lists: content owners and AI builders
- If you own content: audit your crawler traffic now, before the defaults flip. Decide deliberately - block, allow, or monetise - per bot category, and treat AI licensing as a nascent revenue line worth a pricing conversation. If your site runs on Cloudflare's free tier, the September change happens to you automatically; make it a choice instead.
- If you build AI products or agents: your data-supply chain just acquired a deadline and a price tag. Inventory which sources your training refreshes and agent workflows depend on, budget for licensed access where it is critical, and make sure your crawlers are cleanly segmented by purpose - transparent, single-intent bots are exactly what this regime is designed to reward.
- For marketers and strategists, the deeper shift is that visibility inside AI answers is becoming a negotiated, sometimes paid, channel rather than a free by-product of SEO. Build that assumption into 2027 content and distribution plans, and watch the Ceramic.ai and You.com pilots as the early template for how content gets compensated in an answer-engine world.