You updated your WAF rules to block PerplexityBot and Perplexity-User. You checked the server logs and confirmed the requests were dropped. Yet Perplexity still answers specific questions about your restricted content. This is the half-solution trap: you fixed the visible half of the crawling problem, not the whole thing. The missing piece is an undeclared, stealth crawler that impersonates a standard Chrome 124 browser on macOS. It rotates IPs and ASNs to evade static blocks. The proof lies in the traffic gap: Cloudflare’s data shows Perplexity’s declared bots handle 20–25 million daily requests, while this hidden channel generates an additional 3–6 million per day. If you only target the declared user agents, you leave this second channel wide open. Your perplexity indexing exposure remains active even after the official block is in place.
The two crawlers Perplexity declares and the one it does not
Perplexity operates through two declared user agents—PerplexityBot and Perplexity-User—accounting for 20–25 million daily requests. A third, undeclared channel generates 3–6 million daily requests while masquerading as a standard browser. This hidden layer is the missing piece in most current indexing strategies. The table below clarifies the distinction between declared traffic and stealth activity.
| Crawler | User-Agent string | Daily request volume |
|---|---|---|
| Perplexity-User | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) |
20–25M |
| Stealth Crawler | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 |
3–6M |
To confirm if this hidden channel is hitting your infrastructure, search your server or WAF logs for the exact stealth fingerprint: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36. 
This impersonation is not static. The stealth crawler rotates through IP addresses outside Perplexity’s official range and switches between different Autonomous System Numbers (ASNs). This rotation triggers specifically after a restrictive robots.txt policy or a network-level block. A static IP-blocklist remains ineffective because source addresses are constantly changing. When you attempt to fix perplexity scraping by only targeting declared agents, you address only the visible half of the traffic. The undeclared crawler continues to fetch your restricted content. Your ai crawler blocked state is incomplete, and indexing exposure persists.
How Cloudflare confirmed the ai crawler blocked claim was only half true
To test whether Perplexity’s content was truly cut off by blocking declared agents, Cloudflare set up a controlled experiment. They purchased fresh domains that had never been indexed by any search engine, such as testexample.com and secretexample.com. These sites were configured with a strict robots.txt file that disallowed all automated access. They also deployed WAF rules specifically targeting the two known Perplexity user agents.
The results were surprising. When Cloudflare queried Perplexity AI about the content on these restricted test domains, the system still returned detailed and accurate answers. This confirmed that while declared bots were blocked, a separate, non-declared channel was still successfully fetching pages. The assumption that an “ai crawler blocked” status meant total exclusion was, in this case, only half true.
Detection of this hidden traffic relied on more than just user-agent parsing. Cloudflare fingerprinted the stealth crawler using a combination of machine learning and network signals, including IP and ASN patterns and timing data. This approach highlights why simple rules based solely on user-agent strings are structurally insufficient. A rotating crawler that impersonates a standard browser cannot be reliably stopped by static text matching alone. It can switch its identity and network path to evade detection.
Once the stealth crawler was identified and blocked, a clear degradation signal appeared. Perplexity continued to answer questions about restricted content, but responses were noticeably less specific. Answers relied on data from other websites rather than the original source, lacking precise details present before the full block was applied. For site owners, this change in answer quality serves as a practical way to verify that their comprehensive fix for perplexity scraping is working.
Why a complete fix for perplexity scraping matters for AI answer quality
If the business goal is to keep proprietary content out of AI-generated answers, a partial block that lets a stealth crawler through is cosmetic, not functional. When the hidden channel remains active, the model still has access to source material. It can cite specific details, dates, or figures from your site in its responses. In this scenario, the “fix” for perplexity indexing is effectively invisible. The content is still in the context, and the output is unchanged.
The standard for responsible crawling
Well-behaved operators follow established norms. The RFC 9309 standard outlines expectations for web robots, including transparent identification and adherence to robots.txt directives. Cloudflare’s Verified Bots program enforces similar criteria, requiring declared IP ranges and separate agents for different activities.
| Criterion | Compliant Behavior (e.g., OpenAI) | Non-Compliant Behavior (Stealth) |
|---|---|---|
| Identity | Declared, transparent user agent | Impersonates standard browser (Chrome) |
| Network | Uses declared IP ranges | Rotates IPs outside official range |
| Blocking | Stops crawling when blocked | Circumvents via ASN rotation |
OpenAI’s ChatGPT-User agent demonstrates the expected standard. When Cloudflare tested this agent against a disallowed domain, it fetched the robots.txt file, recognized the directive, and stopped crawling immediately. No other user agents from that provider were observed attempting to bypass the block. The Perplexity stealth crawler, by contrast, is the exception. It does not stop. It rotates. This distinction matters because it changes the security model from “allowlist management” to “intrusion detection.”
Impact on AI answer quality
When a brand is only partially blocked, it ends up in AI answers sourced from secondary channels. These might be third-party mirrors, old press clips, or forum discussions. These sources are uncontrolled. They may be outdated, incomplete, or inaccurate. The brand loses the ability to present its narrative as intended.
A full, verified block changes this dynamic. It forces the AI to either exclude content entirely or rely on verified, high-quality sources that the brand approves. This keeps the brand’s narrative clean or removes it from the equation entirely, on the owner’s terms. The goal is not just to stop the crawler. It is to control how the brand is represented in the AI landscape.
The opt-in path
For businesses that prefer controlled visibility over total exclusion, the answer is not to ignore the stealth crawler. It is to declare what is safe to index. An llms.txt setup allows a site to explicitly list which paths and content types are approved for AI consumption. This active declaration works alongside the ai crawler blocked rules for unauthorized access. Relying on the declared bot to be the only reader is a passive strategy that leaves the back door open. Active declaration closes it.
The difference is control. Passive blocking assumes the crawler will respect the boundary. Active management ensures the boundary is respected, regardless of the crawler’s identity.
Perplexity indexing questions: what to check in your own logs
You have blocked the known agents. You confirmed the block in your WAF. Yet your content still surfaces in AI answers. Here is what that actually means for your logs.
I blocked the declared bots. Why is my content still visible?
A separate, undeclared crawler is still fetching your pages. The two declared agents, PerplexityBot and Perplexity-User, represent only part of the total traffic. A hidden channel, impersonating a standard browser, continues to operate in the background, rendering your initial block ineffective.
How do I spot the stealth crawler in my logs?
Search for the user-agent string Chrome/124.0.0.0 running on macOS. This traffic typically originates from IP addresses outside Perplexity’s official range. You will often see these requests arrive in bursts immediately after you implement a robots.txt disallow or a network block.
Can I block it without a paid WAF?
IP and ASN blocklists offer only a partial, fragile mitigation. The stealth crawler rotates through different autonomous systems to evade static lists. A more durable approach is a fingerprinting layer that uses machine learning and network signals, or a managed rule specifically configured to catch stealth AI crawlers.
What if I want controlled indexing instead?
Use an llms.txt file to declare exactly which paths and content types are safe to read. Pair this with an explicit allow rule for the declared user agents. This llms.txt setup gives you precise control over visibility, acting as the positive counterpart to the opt-out block.
Blocking Perplexity indexing is no longer a single-rule task. The split between declared and undeclared crawlers means that a half-block often creates a false sense of security rather than a genuine defense. The practical next step is to verify in your own logs whether the stealth channel is reaching you. Then decide deliberately between a full block and an opt-in via an llms.txt setup. Your current AI-crawler rules were built to stop the crawler Perplexity declares. Or were they built to stop the one it does not?
