Launched in private beta in July 2025 and folded into the AI Crawl Control console, pay per crawl gives publishers a third option beyond "allow" and "block": charge. The site owner sets a price; verified AI crawlers that agree to pay get HTTP 200 and content, others get a 402 with payment terms.
It reframes the crawler standoff as a market: AI companies get legitimate access to content for training, search and agents; publishers get paid per request instead of trading traffic for nothing as zero-click answers grow. Stack Overflow was an early named adopter, and the model was later generalized by Cloudflare's Monetization Gateway.
robots.txt is a voluntary allow/deny signal. Pay per crawl is enforced at Cloudflare's edge and adds a price: crawlers that pay get access, crawlers that don't get a 402 — turning crawl access into revenue rather than an all-or-nothing choice.