Does blocking Google stop Gemini from reading your site? The answer is yes, and no.
The block is real, but it does not do what most site owners expect. A 2026 clarification reshaped the conversation: the Google-Extended entry in your robots.txt is not a crawler visiting your pages. It is a control token.
This distinction reframes how we think about Gemini visibility. Blocking this token does not stop a fetch; it disables a downstream use of data Google has already collected. The entire premise of blocking shifts from preventing access to managing consent.
Google-Extended is a control token, not a crawler
The Google-Extended bot is not a crawler in the traditional sense. It does not visit your website to collect data. Instead, it functions as an opt-out control token within your robots.txt file. When you see this user-agent, it is not indicating a new entity fetching your pages; it is signaling an instruction about how Google should handle data already gathered by the standard Googlebot.
This distinction is crucial for understanding blocking AI crawlers effectively. Regular Googlebot performs the actual crawling and indexing for search. The Google-Extended entry serves a different purpose: it tells Google not to use that already-crawled content for training the Gemini model or grounding Gemini visibility in AI Overviews. It is a downstream switch, not an upstream gate.
Therefore, the word “block” is imprecise here. You are not stopping a fetch; you are disabling a specific use of data Google has already collected. The content remains on the web, and Google still sees it for search purposes, but it is excluded from the AI training pipeline.
| Feature | Purpose | Affected Products | Risk of Blocking |
|---|---|---|---|
| Googlebot | Core search indexing and ranking | Google Search, News, Maps | Critical: Removes site from search |
| Google-Extended | AI training and grounding opt-out | Gemini, AI Overviews | Low: Preserves search visibility |
Implementing the correct syntax
To apply this rule, you only need a single line in your robots.txt file. The correct syntax is:
User-agent: Google-Extended
Disallow: /
This rule is safe to deploy. It will not impact your organic search rankings because it does not interfere with the core search crawler. You are simply opting out of Generative AI indexing for training purposes while retaining full visibility in standard search results. This makes it a low-risk adjustment for publishers who want to control their data’s role in AI systems without sacrificing their SEO foundation.
Which Gemini surfaces a robots.txt rule actually reaches
The distinction between what a site blocks and what Gemini ignores becomes clear when we look at the specific fetchers Google has named. In 2026, Google clarified that several of its Gemini-related agents act on direct, real-time user requests. These include Google-Agent, Gemini Notebook, and Google-Read-Aloud. You will also see references to the former Google-NotebookLM. Because these tools operate at the moment a user types a query, they do not check your robots.txt file before fetching a page. Their behavior is governed by user intent, not site policy.
This leads to a critical split in how your content is processed. Training-time content is governed by the Google-Extended opt-out. If you have blocked the Google-Extended bot, your site’s data is excluded from the models powering Gemini’s general knowledge. However, retrieval-time fetches are a different mechanism. When a user explicitly asks Gemini to summarize a URL or read a document, the system bypasses the training exclusion. It treats your site as a live source of information, fetching the page in real-time to answer that specific request. Blocking the Google-Extended bot does not create a barrier for these on-demand actions.
Practical implications for site owners
You need to understand what a block actually achieves in this context. By blocking the Google-Extended bot, you remove your content from Gemini’s trained knowledge base. This also prevents your site from being used to generate summaries in AI Overviews. If you see your brand missing from the general knowledge base of Gemini, this is the likely cause. However, you must accept that this control has limits. You cannot stop Gemini from visiting your live pages when a user asks it to do so. If a user shares your link with the model, the bot will fetch it. The block affects how Gemini learns, not how it retrieves.
Affected vs. unaffected surfaces
To visualize this, we can look at the two sides of the Gemini ecosystem. On one side, you have the surfaces that respond to your robots.txt rules. These include the underlying models that power general answers and the AI Overviews in search results. Blocking Google-Extended successfully removes your content from these layers. On the other side, you have the interactive, user-triggered tools. Features like the Notebook and Read-Aloud functions are unaffected by your block. They will continue to access your site if a user prompts them to do so. This means your strategy for blocking AI crawlers only controls the background training, not the foreground interactions. It is a partial control, not a total exit. For teams focused on Generative AI indexing, this nuance is essential. You are not hiding your site from Gemini; you are simply opting out of its long-term memory. The line between a static index and a live retrieval tool is where your control ends. Understanding this boundary helps you set realistic expectations for your digital presence in an AI-driven environment. You decide what the model knows, but you cannot always decide what it fetches on demand.
The Googlebot mistake that hides behind a Google-Extended rule
The most frequent error in this space is not a strategic choice but a typo. A site owner intends to block the Google-Extended bot, typing User-agent: Google-Extended followed by Disallow: /. However, a single slip of the finger changes the string to User-agent: Googlebot. The result is not a gentle opt-out from AI training; it is the complete removal of the website from Google Search. Since the two names differ by only one word and appear in the same robots.txt file, this confusion is all too common. Once the wrong rule goes live, organic traffic does not drop—it stops. Search Console will show a rapid decline in crawl stats, and pages will vanish from the index within days, effectively de-indexing the entire property.
To prevent this, verify the exact user-agent token in your robots.txt file before and after any edit. Confirm via Search Console that your site is still being crawled and that no new coverage errors appear. This simple check distinguishes between an intentional Generative AI indexing decision and an accidental catastrophic exclusion. This high-risk asymmetry is why many teams default to allowing Google-Extended: blocking it is a harmless opt-out that preserves search visibility, whereas a mistake involving Googlebot is irreversible in the short term. The risk of de-indexing far outweighs the minor loss of Gemini visibility, making caution the more prudent path for most brands.
Choosing a Generative AI indexing stance that fits your content
Deciding how to handle Generative AI indexing is less about technical configuration and more than aligning your site’s visibility with your business model. The landscape of blocking AI crawlers offers three distinct strategic postures, each suited to a different type of organization.
Three strategic postures for AI visibility
The first approach is allow all, granting maximum AI visibility. This is ideal for content-heavy publishers who value being cited in AI responses, as broad access signals trust and relevance. The second posture is allow search and block AI training. By permitting Googlebot but blocking the Google-Extended bot, you retain organic search traffic while opting out of Gemini’s training data. This selective protection suits premium or membership sites that need to keep gated content out of AI models. The third option is partial access, where you block specific path groups like /premium/ while allowing public sections like /blog/. This hybrid model is well-suited for SaaS or e-commerce brands that want to balance broad reach for general information with strict protection for proprietary or paid content.
Beyond Google: The broader AI crawler landscape
It is a common misconception to treat Google-Extended as the only variable in this equation. In reality, blocking AI crawlers involves a broader ecosystem. Tools like GPTBot from OpenAI, ClaudeBot from Anthropic, and PerplexityBot operate independently of Google’s systems. A rule that stops the Google-Extended bot from training Gemini does not automatically block these other agents. To craft a cohesive strategy, you must define your stance for each major AI player. Ignoring this multi-crawler reality can leave your content partially exposed to models you intended to exclude.
A decision framework for your brand
There is no single correct answer for every business. Instead, use a simple decision framework to determine your stance. Ask yourself: do we value AI-referenced traffic and citation potential, or do we prioritize strict content control and protection of intellectual property? If your brand relies on being a source of general knowledge, allowing access to major crawlers may be the right move. If your value proposition is gated, exclusive content, a more restrictive approach is likely necessary. Ultimately, your Generative AI indexing strategy should reflect where you stand in this trade-off between reach and control.
The 2026 clarification shifts the focus from a binary choice to a layered one. “Blocking” no longer means erasing your site from Gemini; it means deciding which part of Google’s AI stack you are willing to support. One token now controls training, while a separate set of user-triggered fetchers manages retrieval. This split forces a harder question: is the current robots.txt vocabulary precise enough to distinguish between a passive opt-out and an active, user-initiated request for your content?
