For publishers, the current model for managing AI bots is strictly binary: allow access or block the crawler entirely. This all-or-nothing approach offers no middle ground, forcing a choice between open content and zero exposure. However, a third path is emerging. Instead of a simple gate, the edge infrastructure can now demand payment before granting access. This shift moves the conversation from AI bot blocking to a new form of crawlers control where access is priced per request. The technical backbone of this change is the revival of the HTTP 402 Payment Required status code. It transforms the response into a transactional signal, not just a denial. As this mechanism moves from theory to practice, the focus shifts to how leading edge providers implement it. We examine how Cloudflare structures this paid access layer and compare it against Fastly’s current approach to bot filtering and content discovery.
The three-option model for AI crawler management
Traditional AI crawler management forced a binary choice: allow bots or block them. The new pay-per-crawl model introduces a third path: charge.
From binary to ternary control
Publishers now have three distinct options for specific crawlers: Allow, Charge, or Block. This shift moves AI crawler management away from a simple on/off switch. The new option anchors on the specific term pay-per-crawl, enabling publishers to monetize access rather than just denying it.
Merchant of Record and billing
Cloudflare acts as the Merchant of Record for this system. It handles billing aggregation and payouts to publishers. This removes the need for individual, one-off deals with each crawler operator. Publishers set a flat, per-request price for their site, and the platform manages the financial flow.
Operational distinction: 403 vs 402
A hard block uses HTTP 403 Forbidden and returns no content. A charge decision uses HTTP 402 Payment Required. If a publisher chooses to charge a crawler without an existing billing relationship, the function is equivalent to a block—no content is returned. However, the 402 status signals that a future paid relationship is possible, unlike the definitive rejection of a 403.
Per-crawler bypass
Publishers can exclude specific crawlers from the charge rule. This per-crawler bypass allows for free access to certain bots or facilitates separately negotiated content partnerships. It provides granular control over how different AI agents interact with the site, adding flexibility to the broader crawlers control strategy.
How pay-per-crawl enforces AI bot access at the edge
AI crawler management shifts from static rules to active transactional logic at the network perimeter. Cloudflare implements this using two distinct access flows that determine how payment intent is declared. The reactive flow is discovery-first: a crawler hits a paid URL and receives an HTTP 402 response containing a crawler-price header. The crawler then retries the request with a crawler-exact-price header, signaling agreement to pay the configured rate. The proactive flow is intent-first: the crawler includes a crawler-max-price header in the initial request. If this amount matches or exceeds the site’s price, the request succeeds with a 200 OK response; if it falls short, the system returns a 402.
Cryptographic Identity and Request Validation
To prevent unauthorized access, the system relies on HTTP Message Signatures bound to Ed25519 keys. A crawler must generate a key pair and publish the public key in a JWK-formatted directory. After registering this directory URL with Cloudflare, every request must include signature-agent, signature-input, and signature headers. This mechanism prevents spoofing because a competitor or malicious bot cannot masquerade as a registered, paying crawler; the signature is cryptographically linked to the registered key, making impersonation impossible without access to the private key.
Integration and Operational Constraints
The pay-per-crawl rules engine does not replace existing security infrastructure. It runs only after WAF policies and bot-management rules have been applied. This sequence is critical for sites already running a security stack: if a WAF rule blocks a user agent, it never reaches the pay-per-crawl logic. Additionally, the system enforces strict header syntax. Only one price-declaration header (crawler-exact-price or crawler-max-price) may be used per request. Requests containing both, or malformed pricing data, are rejected with a 402 status, ensuring that only valid, authenticated payment intents proceed to the billing layer.
Cloudflare vs Fastly: what the comparison actually covers
Before assessing Cloudflare vs Fastly for AI crawler management, it is essential to acknowledge the sourcing asymmetry. The technical details regarding Cloudflare’s pay-per-crawl mechanism are derived from official product announcements. In contrast, the Fastly side of this comparison relies on general knowledge of their platform capabilities. We do not present these two as evenly sourced or deeply vetted; rather, we map what is currently known to help you evaluate your options.
Cloudflare’s current AI bot access control stack operates in layers. The foundation consists of Web Application Firewall (WAF) rules and standard bot management features. Above this security base sits the new pay-per-crawl layer, which handles the financial transaction of access. This integration means that AI crawler management does not replace your existing security posture but extends it.
Fastly offers a different set of capabilities. Their platform is known for managed challenges, WAF rules, and general bot filtering. They are also recognized for handling the llms.txt file at the edge. However, specific details regarding a native pay-for-access mechanism for AI crawlers are not defined in the current official documentation provided for this analysis. You should treat Fastly’s specific AI-bot features as general knowledge and verify them against their current technical documentation.
| Decision Dimension | Cloudflare | Fastly |
|---|---|---|
| Pay-for-access mechanism | Available (private beta) | General knowledge — verify against current documentation. |
| Crawler authentication | Ed25519 Web Bot Auth signatures | General knowledge — verify against current documentation. |
| Security policy integration | Runs after WAF and bot rules | General knowledge — verify against current documentation. |
| llms.txt support | Independent of pay-per-crawl | Handled at the edge |
This comparison is not a final verdict. As both vendors ship new AI-bot features, the landscape shifts. We recommend you treat this framework as a living document, revisiting it as new capabilities emerge to ensure your strategy for AI crawler management remains aligned with the latest technical realities.
Practical questions on AI crawler management and llms.txt
Granular crawler control
Can you charge one crawler while letting another through for free? Yes. The per-crawler bypass allows you to exclude specific bots from the domain-wide flat price. This is useful if you want to grant free access to certain sources or honor separately negotiated content deals without disrupting the broader charge rule.
The role of HTTP 402
What happens when a registered crawler hits a paid resource without the right header? The edge returns 402 Payment Required with a crawler-price header. The crawler then retries using either crawler-exact-price (agreeing to the exact amount) or crawler-max-price (stating its ceiling). This two-step exchange defines the reactive access flow, keeping the transaction logic transparent within the request lifecycle.
Where llms.txt fits
llms.txt is a file-based mechanism where publishers list content available to AI systems. It operates at the discovery layer and is independent of the 402 payment mechanism. Fastly is known to handle llms.txt at the edge, whereas Cloudflare’s pay-per-crawl system does not rely on it for access enforcement.
Security integration and release status
Does this replace WAF or bot management? No. Pay-per-crawl runs only after existing WAF and bot-management policies are applied. If a crawler is blocked upstream, it never reaches the charge logic. It is an additional layer, not a replacement. Currently, the feature is in private beta; while the architecture is defined, pricing tiers and general availability are still evolving.
AI crawler management in practice
For teams implementing AI crawler management, the key takeaway is that control is layered. You can manage access, payment, and security independently, allowing for nuanced strategies that go beyond simple blocking.
Pay-per-crawl remains an early experiment, a glimpse of an agentic-web future where AI agents carry budgets to negotiate content access programmatically—a scenario still years away from standard practice. As training and inference workloads scale, the current binary model of free access or total exclusion may not hold. Will publishers find value in granular crawlers control, or will the friction of paid access outpace the demand?
