Question
What is Google-Extended?
Google-Extended is a robots.txt token, introduced by Google, that lets you control whether your content is used to train and ground Gemini models — without affecting Google Search crawling or ranking, which Googlebot still handles.
Why it exists
Site owners wanted a way to stay in Google Search while opting out of generative AI training. Google-Extended separates those two consents into different robots.txt tokens so you do not have to trade ranking for AI control.
Key facts
- It is a policy token, not a distinct crawler with its own IPs or user-agent string.
- Blocking it has no effect on Google Search indexing, ranking, or normal crawling by Googlebot.
- It governs Gemini app and Vertex/Gemini API training and grounding use of your content.
How to set it
User-agent: Google-Extended
Disallow: / # opt out of Gemini training/grounding
# (omit or use Allow: / to opt in)Practical examples
A publisher that wants Google Search traffic but does not want article text used for Gemini grounding can block Google-Extended while still allowing Googlebot. A SaaS company that wants help docs to appear in AI answers may leave Google-Extended allowed for public docs, but block account, admin, and pricing-experiment paths separately.
The important check is not whether a request shows a separate Google-Extended user agent in logs. Treat it as a robots policy decision. Verify that /robots.txt returns 200, that the rule is not overridden by a broader wildcard block, and that Googlebot can still fetch your public source pages.
Page-type decisions
Use page type, not site-wide habit, to decide the rule. Public docs, product explainers, FAQs, and comparison pages are usually safe to allow when your goal is answer visibility. Account pages, private dashboards, paid lesson bodies, internal search results, and experimental pricing pages should stay blocked or unlinked regardless of AI policy.
- Allow: evergreen documentation, product pages, public course overviews, support FAQs, and case studies you want cited.
- Block: private portals, checkout flows, member-only lessons, internal search URLs, staging hosts, and thin campaign variants.
- Review manually: licensed editorial archives, partner-only reports, and pages containing user-generated or regulated content.
Safe policy pattern
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /private/
Disallow: /account/
Allow: /docs/
Allow: /blog/If your goal is search visibility, avoid accidentally blocking Googlebot while editing AI-training rules. Keep Search crawling, AI grounding, and private-path protection as separate checks.
Related pages: should I allow Google-Extended, AI crawler user agents.