What Is Google-Extended — and Should You Block It?

Google-Extended is the most misread line in AI-era robots.txt files. Operators paste it into block lists expecting to disappear from AI Overviews, or leave it alone fearing a ranking penalty, and per Google's own documentation both expectations are wrong. Since August 31, 2026 the confusion has a cleaner answer: Google documents a separate Search Console setting for AI Overviews and AI Mode, and says that setting does not touch training. This page states what each control governs, quotes the documentation for each boundary, and gives the decision the way we make it on our own builds.

What is Google-Extended?

Google-Extended is a robots.txt control token, not a crawler. Google's crawler documentation describes it as "a standalone product token" that publishers use to manage whether content Google crawls from their sites may be used for "training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini" and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI (Google's common crawlers, page updated 2026-07-14).

The mechanical detail that confuses everyone: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings." No bot named Google-Extended will ever fetch a page from you. Googlebot and its siblings do the fetching exactly as before; the token only changes what Google permits itself to do with what it fetched.

That makes Google-Extended different in kind from GPTBot or ClaudeBot, which are real crawlers you can count in logs. It is one line of policy, addressed to one company, enforced entirely inside that company's pipeline; the full roster of actual bots is in the complete AI crawler list.

What does disallowing Google-Extended actually stop?

Two documented uses: training future Gemini models, and grounding answers in Gemini Apps and Grounding with Google Search on Vertex AI. Google's Search Console help adds that the token is also how you limit training of the models behind Search's AI features; neither use touches inclusion or ranking in Search. Everything else operators attribute to the token is projection. The boundary map, per Google's crawler, AI-features and Search Console documentation as read on 2026-10-14:

Google's AI-features page draws the same line from the other side: it points publishers to Google-Extended "to limit AI training and grounding in some of Google's other systems," meaning systems other than the Search features that page is about (Google).

How is Google-Extended different from Search Console's AI control?

Google-Extended governs training and Gemini grounding; the Search generative AI control, which Google rolled out to all websites on August 31, 2026, governs whether your links and content appear in AI Overviews, AI Mode and Discover's generative AI features. Google's help page for the control says it "doesn't affect AI training; to limit training of the models used to generate responses in Search generative AI features, use Google-Extended" (Search Console Help).

Which Google control does what, per Google's documentation as read on 2026-10-14:

Read together, the two documents answer the question publishers actually have. A site that wants out of Gemini training sets Google-Extended. A site that wants out of AI Overviews and AI Mode sets the Search Console control. A site that wants both sets both, and per Google neither one moves its classic rankings.

Does blocking Google-Extended remove you from AI Overviews?

No. Google-Extended governs Gemini training and grounding in Google's other systems, while AI Overviews and AI Mode are Search features: eligible pages must be indexed and snippet-eligible, and the site must be included in Search generative AI features in Search Console, which is the default (Google). The levers for those features are the Search Console control and the snippet controls in Google's robots meta tag reference.

So a site that disallows Google-Extended and then sees itself cited in an AI Overview has not caught Google ignoring robots.txt. It has demonstrated the documented boundary: AI Overviews are a Search feature, fed by Search's index, governed by Search's controls. The two generative surfaces and their shared rules are unpacked in what Google AI Mode is.

How do you block Google-Extended in robots.txt?

One group, two lines, using the token verbatim:

User-agent: Google-Extended
Disallow: /

Partial rules work the same way as for any token. Google's own example opts out a directory while keeping one page in: user-agent: Google-Extended, then allow: /archive/1Q84 and disallow: /archive/. Two operational notes. First, there is nothing to verify in your logs afterward: no user agent changes behavior, because none exists, so the file itself is your only record. Second, under RFC 9309 robots.txt rules "are not a form of access authorization"; the token is a published instruction to one company about permitted use.

If a CDN writes part of your robots.txt, read what it publishes. Cloudflare's "Disallow AI Training" setting, introduced September 15, 2026, publishes a no-training preference while keeping Googlebot crawling for search, and Cloudflare names Google-Extended as Google's training opt-out (Cloudflare). Its "Block" setting now stops Googlebot entirely; the setting-by-setting map is in what Cloudflare's AI crawler controls do.

Should you block Google-Extended?

Decide it as a training-posture question, because per Google's documentation that is the only thing at stake. Blocking GPTBot or ClaudeBot at least changes observable crawl traffic; blocking Google-Extended has no stated cost to Search inclusion, ranking, or AI-feature eligibility, and no observable effect at all outside Google's own systems. That makes it the cheapest "no" a publisher can say to Gemini training.

On our 3 production builds we leave it open [our data]: our sites earn from being visible and accurately represented wherever answers get assembled, so Gemini training and grounding access is consistent with how we make money. Against our own interest here: if you license content, sell subscriptions, or otherwise treat your archive as the asset, disallowing Google-Extended is the rare opt-out with no documented visibility bill attached, and we would take it in your position. The same job-by-job logic for every other token is in the robots.txt decision framework.

One honest caveat closes the decision. The token governs exactly what Google's documentation says it governs, and that scope is Google's to define, which is why we re-verify the crawler docs quarterly rather than trusting any summary, including this one, indefinitely. What being retrievable and citable is actually worth, the reason most sites leave every door open, is the subject of our generative engine optimization guide.