You likely do not need a separate rule for “Copilot” in your robots.txt file. That user agent does not exist. Microsoft Copilot does not operate a standalone crawler; it pulls its answers entirely from the Bing index, which is built exclusively by a single agent: Bingbot. If you are searching for a way to exclude Copilot AI or looking for a specific Bing AI crawler identifier, you are looking in the wrong place. The only directive that controls Copilot access is the one governing Bingbot.
The misconception that access is a simple on/off switch is why many brands remain invisible in generative answers without understanding why. In reality, the system offers gradations. While blocking Bingbot in robots.txt cuts you off completely, you can keep your pages indexed for standard search while controlling how much content appears in AI summaries. This article outlines the hidden meta tags that allow you to fine-tune that visibility, ensuring your brand appears in the answers users actually read.
Bingbot Is the Only Gate to Copilot
When you search for the Copilot user agent in your server logs, you will not find one. This absence is intentional. Microsoft does not run a separate crawler for its AI assistant. Instead, every answer generated by Copilot is grounded in the Bing index, a database built exclusively by Bingbot.
This distinction matters because many site owners confuse the different Microsoft bots that visit their servers. While Bingbot has full access to your content for indexing and AI grounding, two other agents serve entirely different functions. Blocking the wrong one can degrade user experience without affecting your AI visibility, while blocking the right one silences your brand in generative answers.
Understanding this architecture is the first step in managing your Bing AI crawler presence. It clarifies that access to Copilot is not controlled by a new, separate permission layer, but by the same rules you already apply to Bing Search. The difference lies in how that content is used after the crawl.
Identifying the Right Microsoft Agents
To configure your Copilot robots.txt correctly, you must distinguish between the three primary Microsoft agents. Each has a specific purpose and a distinct consequence if blocked. Confusing them can lead to unnecessary visibility loss or, conversely, missed opportunities to protect sensitive data.
Here is how the three agents function and what happens if you block them:
| User Agent | Primary Purpose | Effect of Blocking |
|---|---|---|
Bingbot |
Builds the Bing index used by Search and Copilot | Removes the site from both Bing Search and Copilot answers entirely. |
MicrosoftPreview |
Generates link previews for Microsoft products | Degrades link previews in Microsoft surfaces; links remain functional. |
AdIdxBot |
Validates landing pages for Microsoft Advertising | Causes ad quality reviews to fail, potentially pausing campaigns. |
Note that blocking MicrosoftPreview or AdIdxBot does not impact how Copilot generates answers. If you need to exclude Copilot AI from seeing specific content, the only relevant lever in your robots.txt is the Bingbot directive. The other two agents operate on a separate pipeline that does not feed into the generative search model.
Configuring Bingbot for AI Answer Access
Getting the syntax right in your robots.txt file is less about blocking and more about surgical precision. Bing honors standard Allow and Disallow directives, but how you write them determines whether your fresh content gets indexed quickly or gets stuck in the queue.
The most common mistake in this configuration is matching the entire user-agent string. You might see a full identifier like Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z. If you hard-code that into your web server rules, it will eventually break. The Chrome version numbers and Safari tokens in that string rotate frequently, meaning a rule that works today might stop matching the crawler next month. The reliable approach is to match simply on the bingbot token. This allows your server to recognize the Bing AI crawler regardless of the browser emulation tokens attached to it.
When structuring the file, you want to allow general crawling while keeping private areas off-limits. A practical configuration looks like this:
User-agent: bingbot
Allow: /
Disallow: /account/
Disallow: /cart/
Disallow: /private/
This structure ensures that public content remains accessible for AI answer generation, while user-specific or transactional paths are restricted. It is a standard pattern for any major crawler, but it is particularly critical here since Bing uses these paths to build the specific context for Copilot’s answers.
Managing Crawl Speed Without Crawl-delay
Many developers reach for the Crawl-delay directive as a quick way to throttle the Bing AI crawler and protect server resources. While Bing does respect this directive—unlike Google—it is generally not the recommended method for controlling traffic. Adding a delay of 10 or 15 seconds to every request can significantly slow down the indexation of fresh content. If you publish an article, that delay applies to every page Bing touches, which can lead to lagging visibility in generative search results.
Instead, we recommend managing the crawl rate directly through Bing Webmaster Tools. This allows you to set a specific crawl frequency limit for the entire domain without the overhead of per-request delays. You can adjust these settings from a central dashboard if you notice a spike in traffic, keeping your server stable without sacrificing the speed at which your content reaches AI-driven answers.
By handling speed at the tool level and keeping your robots.txt focused on path permissions, you maintain a clean configuration that is easy to audit and less prone to version-related errors.
The Real Control: nocache and noarchive Meta Tags
Robots.txt files operate on a binary logic: a site is either accessible or blocked. However, when Bing integrated specific directives into its index in September 2023, it introduced a more granular system for managing how content is utilized in AI answers. While the initial crawl access remains a yes-or-no decision, the meta tags now provide the nuance needed to control the depth of that usage. This two-tier approach allows site owners to decide not just if Copilot can see a page, but exactly what it can do with the information it finds.
The Granularity of AI Answer Usage
nocache and noarchive are the two primary directives that bridge the gap between simple index inclusion and specific AI behavior. They function as a middle ground, offering control without removing the site from search results entirely. This distinction is critical for brands that want to remain visible in Bing’s standard search listings but wish to manage their presence in generative AI responses. By using these tags, we can influence whether our content serves as a cited source or merely a reference link.
Defining nocache
The nocache directive limits the scope of what Copilot can use from a specific URL. When implemented via <meta name="robots" content="nocache">, it restricts the AI to referencing only the page’s URL, title, and snippet. In practical terms, this makes the site linkable but unquotable. The AI cannot pull direct excerpts or paragraphs from the page to construct its answer, but it can still point users toward the resource. This is often a strategic choice for pages containing sensitive or proprietary information where a link is desired, but detailed extraction is not.
Defining noarchive
In contrast, the noarchive directive is more restrictive regarding AI generation. Defined by <meta name="robots" content="noarchive">, this tag excludes content from Copilot answers entirely. The key difference here is that standard Bing search listings are preserved. Users can still find and click on the page through the search engine, but the AI will not cite or summarize the page’s content when generating its responses. For those looking to effectively exclude Copilot AI from using specific data, this is the definitive control, ensuring the content remains outside the generative answer space while staying indexed for traditional search.
Hidden noarchive Tags in CMS Presets
A common pitfall in legacy CMS setups involves security presets that have silently added noarchive tags site-wide. These configurations were often deployed years ago to prevent search engines from caching pages, a standard practice for protecting sensitive data or ensuring freshness. Over time, these rules become embedded in templates and theme files, invisible to content managers who focus on publishing rather than code structure.
The impact of this oversight is significant. A site owner may believe their content is fully visible, while in reality, the platform is opting the site out of AI answers without their knowledge. This effectively silences the brand in generative search before the reader even knows the mechanism exists. The content remains available in traditional search results, but the nuance of being excluded from the AI layer means the brand is missing out on the new primary channel for user interaction.
Auditing for Accidental Exclusion
We recommend auditing your website templates and underlying code to verify the current state of these meta tags. It is crucial to determine whether any exclusion is intentional rather than an accidental leftover from a security hardening initiative. Look for global template variables or header injection scripts that append noarchive to every page. If the goal was to prevent caching of sensitive areas, apply the restriction specifically to those paths rather than site-wide. Ensuring the exclusion is deliberate is the first step to restoring your visibility in the Bing AI crawler ecosystem.
Bingbot vs GPTBot and Security Verification
A common point of confusion involves the relationship between GPTBot and Microsoft Copilot. GPTBot is OpenAI’s training crawler, and it has no operational link to Microsoft’s retrieval process. Decisions regarding GPTBot in your robots.txt file are independent of Copilot visibility. Blocking GPTBot does not exclude your site from Copilot answers, as Copilot relies exclusively on the Bing AI crawler (Bingbot) for its index. You can block one, allow the other, or do both without cross-interference.
Why User-Agent Strings Are Not Enough
While identifying the crawler is the first step, trusting the user-agent string alone is a security risk. Scrapers frequently spoof legitimate identifiers to bypass basic filters. A request claiming to be Bingbot could easily be a data-harvesting script. To protect your infrastructure, verification must go beyond the string.
We recommend verifying genuine Bingbot traffic using two specific methods:
- Reverse DNS Lookup: Resolve the IP address of the request. A legitimate Bingbot request must resolve to a hostname ending in
search.msn.com. - Forward Confirmation: Perform a forward DNS lookup on that resolved hostname. It must return the original IP address to confirm the match.
Configuring Your WAF for Legitimate Crawlers
Once you have a verified list of legitimate IPs, use Microsoft’s published JSON list of Bingbot IP ranges to configure your Web Application Firewall (WAF). This allowlist prevents bot-management defaults from throttling the actual crawler.
Many edge providers, including Cloudflare, Akamai, and Fastly, apply strict rate limits or JavaScript challenges to bots by default. These rules often treat Bingbot more aggressively than Googlebot. By explicitly exempting verified Bingbot traffic from CAPTCHAs and rate limits, you ensure your content remains accessible for indexing. A sudden drop in logged Bingbot fetch volume is a critical early warning sign that a firewall rule may have accidentally blocked the crawler.
The decision of whether your content appears in AI answers is no longer a simple on/off toggle in a single file. It is a layered system involving crawler identification, index inclusion, and specific meta-tag gradations that work together to define your visibility. A quick look at your current robots.txt and meta tags might reveal if a legacy security setting is accidentally silencing your brand in the next wave of search.
