Google-Extended
Definition: Google-Extended is a robots.txt product token, not a separate crawler, that lets publishers control whether content Google crawls may be used to train Gemini models and to ground answers in Gemini Apps and Vertex AI.
Also called: Google Extended
Updated
A token, not a bot
Google-Extended is easy to misunderstand because it appears in robots.txt like a crawler name. Google says it has no separate HTTP user agent string: crawling is done with Google’s existing user agents, and Google-Extended is a control token read from robots.txt (Google’s crawler documentation). You will never see it in your server logs.
What it controls, according to the same page:
- whether content Google crawls from your site may be used to train future Gemini models, and
- whether it may be used for grounding in Gemini Apps and in Grounding with Google Search on Vertex AI.
What it does not control
Google says Google-Extended does not affect a site’s inclusion in Google Search and is not used as a ranking signal. AI Overviews and AI Mode are part of Search and are built from what normal Googlebot crawls, so blocking Google-Extended leaves you in them.
If your aim is to limit how much of your content AI Overviews and AI Mode use, the controls are the snippet rules nosnippet and max-snippet, which also shorten or remove your regular search snippet.
How to use it
User-agent: Google-Extended
Disallow: /The common mistakes run in both directions. Some sites block Google-Extended expecting to leave AI Overviews, which it does not do. Others block Googlebot itself to stay out of AI features and drop out of Search entirely. The AI crawlers guide lists the right control for each Google product.