Question
Should I allow Google-Extended?
Google-Extended is a robots.txt control for whether your already-indexed content is used to train and ground Gemini. Blocking it does not affect Google Search indexing or ranking. Keep it allowed if you want Gemini to be able to ground answers on your content; block it if you want to opt out of generative training.
What Google-Extended is and is not
- It is: a standalone robots.txt token that governs Gemini app/API training and grounding use.
- It is not: a crawler with its own user-agent — Googlebot still does the actual crawling.
- It does not change how you rank in Google Search or appear in AI Overviews indexing.
How to decide
- Want maximum AI answer surface: allow Google-Extended so Gemini can ground on you.
- Publisher protecting licensed content: block it to opt out of generative use while keeping Search.
- Unsure: allowing is the lower-risk default for visibility-first sites.
Decision table
| Site type | Recommended rule | Reason | Extra control |
|---|---|---|---|
| Public docs, developer tools, support FAQs | Allow | You likely want Gemini users to discover and cite your implementation answers. | Keep private admin and staging paths blocked separately. |
| Local service, SaaS, B2B lead site | Allow public pages | Product, comparison, FAQ, and case study pages are usually meant to be surfaced. | Block account, checkout, CRM, and internal search paths. |
| Publisher with licensed or syndicated content | Review or block | Search visibility may be valuable while generative reuse may be contractually sensitive. | Use path-specific rules for premium archives and licensed sections. |
| Course platform or paid knowledge base | Mixed | Public course landing pages can be visible; lesson bodies may be member-only. | Allow overview pages, block member lessons and download paths. |
| Regulated, user-generated, or partner-only content | Block or isolate | The risk is usually higher than the answer-visibility benefit. | Use noindex/authentication in addition to robots rules where needed. |
Common site scenarios
- Documentation or SaaS help center: allow public docs if your goal is to be cited when users ask Gemini implementation questions.
- News, paid research, or licensed editorial content: block Google-Extended globally or by premium-content path while keeping Googlebot allowed.
- Local service or B2B lead site: allow public service pages, case studies, and FAQs, but block private portals, staging paths, and account pages.
Mixed allow/block example
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Allow: /docs/
Allow: /blog/
Allow: /case-studies/
Disallow: /account/
Disallow: /members/
Disallow: /research/private/This keeps normal Search crawling open while giving Gemini access only to source pages you are comfortable having used for generative grounding.
Concrete examples
SaaS or B2B lead generation site
For a visibility-first SaaS site, allow pages that explain what the product does and answer buying questions. Block customer-only or transactional paths. The business value is in being discoverable for implementation, comparison, and problem-solution questions.
User-agent: Google-Extended
Allow: /features/
Allow: /comparisons/
Allow: /case-studies/
Allow: /questions/
Disallow: /app/
Disallow: /account/
Disallow: /checkout/
Disallow: /admin/Local service business
For a clinic, law firm, agency, repair service, or local operator, the public pages are usually intended to be found and summarized. Keep service pages, FAQs, locations, and reviews open. Block intake forms, appointment dashboards, and private client files.
User-agent: Google-Extended
Allow: /services/
Allow: /locations/
Allow: /faq/
Allow: /case-studies/
Disallow: /booking/
Disallow: /client-portal/
Disallow: /forms/private/Publisher, course, or paid research site
Use a mixed policy. Let landing pages and free explainers remain visible, but block paid lessons, subscriber archives, licensed feeds, and downloadable reports if reuse rights are sensitive.
User-agent: Google-Extended
Allow: /free-guides/
Allow: /course/
Disallow: /course/lessons/
Disallow: /members/
Disallow: /paid-reports/
Disallow: /licensed-content/robots.txt to block generative use only
# Keep normal Google Search; opt out of Gemini training/grounding
User-agent: Google-Extended
Disallow: /
User-agent: Googlebot
Allow: /Visibility-first policy example
For a site that wants answer visibility but still has sensitive sections, avoid a site-wide block. Give Google-Extended access to pages you would be comfortable seeing summarized or cited, and block the parts that should not feed AI systems.
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Allow: /docs/
Allow: /questions/
Allow: /case-studies/
Allow: /use-cases/
Disallow: /account/
Disallow: /checkout/
Disallow: /members/
Disallow: /internal-search/
Disallow: /staging/If your concern is Google Search AI Overviews
Do not use Google-Extended as your main control for Search AI features. Google documents AI features in Search as part of Search, where Googlebot access and preview controls are the relevant levers. If you need to limit what can be shown from a page in Search, review nosnippet, max-snippet, data-nosnippet, or noindex instead of blocking Googlebot by accident.
<meta name="robots" content="max-snippet:120">
<span data-nosnippet>Do not use this sentence in a Search snippet.</span>Confirm it is in place
curl -s https://yourdomain.com/robots.txt | grep -A8 -i 'google-extended'
curl -I https://yourdomain.com/robots.txt
# Then test a sample rule with Google's robots.txt tester or your own parser.
# Do not look for a separate Google-Extended user-agent in access logs.What not to infer from logs
Do not expect to see a distinct Google-Extended crawler in access logs. Googlebot handles crawling; Google-Extended is a policy token. After changing it, confirm the robots rule and then monitor whether Googlebot can still fetch public pages with normal 200 or 304 responses.
Official references: Google crawler docs for Google-Extended and Google Search AI features controls.
Related pages: what is Google-Extended, should I allow GPTBot, AI crawler user agents.