robots.txt for AI bots: copy-paste policies

Blocking GPTBot does not remove your site from ChatGPT; blocking OAI-SearchBot does. Most mistakes in robots.txt for AI bots come from that kind of mix-up, because each AI company runs separate agents for search, for training and for fetching pages on a user’s request, and each one obeys its own line in your file.

This page gives you three complete policies to copy, a reference table of 17 AI bots with what blocking each one does, and the rules that catch people out. Every bot here is one our robots.txt tester checks.

Which policy fits your site?

Policy	Use it when	You keep	You give up
1. Maximize AI citations	You want to be found and cited everywhere	All AI search and assistant visibility	Any say over training use
2. Search yes, training no	You want AI search traffic but not to supply training data	ChatGPT search, Claude, Perplexity, Google AI features, Copilot	Influence over what future models know about you without searching
3. Block all AI	Licensed, paywalled or sensitive content	A training and AI-search opt-out with the major vendors	Citations in AI answers

Our view: policy 2 suits most businesses that sell something. AI search answers send traffic and cite sources; training data does neither directly.

Policy 1: maximize AI citations

# Policy 1: allow all crawlers, including AI search, assistants and training
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

If your current file disallows private paths such as /admin/ or /cart/, keep those rules in the * group. Do not add named AI groups to this policy unless a bot needs different rules, for the reason explained under rule 1 below.

Policy 2: search yes, training no

# Policy 2: allow AI search and user-triggered fetches, opt out of training
User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

What this does, vendor by vendor:

• OpenAI: GPTBot is out; OAI-SearchBot and ChatGPT-User fall under * and stay allowed. OpenAI says each setting is independent (OpenAI crawler documentation).
• Anthropic: ClaudeBot (training) is out; Claude-SearchBot and Claude-User stay allowed. Anthropic says blocking ClaudeBot signals that future material should be excluded from training (Anthropic).
• Google: Google-Extended is a control token, not a crawler. Google says it “does not impact a site’s inclusion in Google Search” and is not a ranking signal, but it does cover grounding in Gemini Apps and Vertex AI as well as Gemini training (Google crawler documentation). That grounding loss is the trade-off.
• Apple: Applebot-Extended is also a token. Apple says pages that disallow it can still appear in search results in Siri, Spotlight and Safari.
• Amazon: Amazonbot may be used to train Amazon AI models. Amazon’s search features use a separate agent, Amzn-SearchBot, which this policy leaves allowed.
• Meta and Common Crawl: meta-externalagent crawls for Meta’s AI training; CCBot builds the public Common Crawl archive that many AI developers train on.

Policy 3: block all AI crawlers

# Policy 3: block AI search, assistant and training crawlers
# Googlebot and Bingbot stay allowed so you remain in Google and Bing
User-agent: *
Allow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: DuckAssistBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: CCBot
Disallow: /

Three gaps this file cannot close:

1. Google’s AI features. AI Overviews and AI Mode use the normal Googlebot crawl, so blocking them in robots.txt means leaving Google Search. Instead, use the Search generative AI control in Search Console. Google says excluding your site there removes its content from AI Overviews, AI Mode and generative features in Discover, is not used as a ranking signal for the rest of Search, and takes effect within one to two days for most content (Google Search Console Help).
2. Copilot. Microsoft’s AI answers draw on Bing, so the only robots.txt lever is Bingbot, which also removes you from Bing.
3. User-triggered fetchers. Perplexity says Perplexity-User “generally ignores robots.txt rules” (Perplexity crawler documentation). Meta says its user-triggered fetcher may bypass robots.txt too. Blocking these takes a firewall rule; Perplexity publishes its IP ranges for that purpose.

Every AI bot, what it does and what blocking it changes

User agent	Operator	Purpose	Type	Follows robots.txt	Effect of blocking
Googlebot	Google	Google Search, AI Overviews and AI Mode	Search	Yes	Removes you from Google Search, AI features included
Bingbot	Microsoft	Bing Search and Copilot	Search	Yes	Removes you from Bing and from Copilot answers grounded in it
OAI-SearchBot	OpenAI	ChatGPT search results	Search	Yes	Out of ChatGPT search answers; navigational links may remain
Claude-SearchBot	Anthropic	Claude search results	Search	Yes	Lower visibility in Claude’s search results
PerplexityBot	Perplexity	Perplexity search index	Search	Yes	Out of the index Perplexity answers from
Applebot	Apple	Siri, Spotlight and Safari search	Search	Yes	Out of Apple search features
DuckAssistBot	DuckDuckGo	DuckDuckGo AI-assisted answers	Search	Yes	Out of DuckAssist answers; DuckDuckGo says the data is not used for training
ChatGPT-User	OpenAI	Opens a page when a ChatGPT user asks	User	May not apply	ChatGPT may be unable to open your page on request
Claude-User	Anthropic	Opens a page when a Claude user asks	User	Yes	Lower visibility in user-directed searches in Claude
Perplexity-User	Perplexity	Live fetch for a user’s question	User	Generally no	Little effect; use a firewall rule
GPTBot	OpenAI	Model training	Training	Yes	OpenAI training opt-out; no effect on ChatGPT search
ClaudeBot	Anthropic	Model training	Training	Yes	Anthropic training opt-out
Google-Extended	Google	Gemini training and grounding (token)	Training	Yes	No effect on Google Search; opts out of Gemini training and grounding
Applebot-Extended	Apple	Apple model training (token)	Training	Yes	No effect on Apple search features
meta-externalagent	Meta	AI training and indexing	Training	Yes	Opts out of Meta’s crawl
Amazonbot	Amazon	Product improvement; may train Amazon AI models	Training	Yes	Amazon search uses Amzn-SearchBot instead
CCBot	Common Crawl	Public web archive used by many AI developers	Training	Yes	Future pages left out of Common Crawl

The full background on each agent is in the AI crawlers guide, and the GPTBot glossary entry covers OpenAI’s training crawler in detail.

Rules that trip people up

1. A named group replaces the * group. A crawler follows only the most specific group that matches its name. If you add User-agent: OAI-SearchBot with Allow: /, your * rules (say, Disallow: /admin/) no longer apply to OAI-SearchBot. Repeat any path rules you still want inside each named group.
2. One file per host. shop.example.com needs its own robots.txt; rules on www.example.com do not cover it.
3. Tokens never appear in your logs. Google says Google-Extended has no separate user-agent string, and Apple says Applebot-Extended does not crawl. Their rules change how content fetched by Googlebot and Applebot is used.
4. Changes are not instant. OpenAI and Perplexity say about 24 hours; DuckDuckGo says 72 hours.
5. Firewalls override robots.txt. If your CDN blocks an AI crawler that robots.txt allows, the block wins and you will not be cited. Anthropic also warns that blocking its IP addresses can stop it from reading your robots.txt, which makes opt-outs unreliable.
6. robots.txt is a request, not access control. Content that must stay private needs authentication, not a Disallow line.
7. llms.txt is not a substitute. It does not grant or deny access to anything. See the llms.txt guide for what it is and is not used for.

How do you test your robots.txt for AI bots?

Paste a URL into the robots.txt tester to see whether each crawler above may fetch it under your live file. The AI visibility checker goes a step further: it also requests the page using several AI crawlers’ user agents, so you can spot a CDN or firewall block that robots.txt does not show. Your server logs remain the definitive record of which bots actually arrive.

For a refresher on the file format itself, see the robots.txt glossary entry.

What to do next

Decide on a policy before you edit anything, and write down why: the choice between policies 1, 2 and 3 is a business decision, not a technical one. Copy the matching block, keep your existing path rules, and test five key URLs with the tester after deploying. If being cited is the goal, the next steps are in how to get cited by ChatGPT and Perplexity SEO.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

No. OpenAI says GPTBot only controls whether your content may be used to train its models. ChatGPT search inclusion is controlled by OAI-SearchBot, and each setting is independent. You can block GPTBot and still be cited in ChatGPT search answers.

How do I block Google's AI Overviews in robots.txt?

You cannot do it with robots.txt without leaving Google Search, because AI Overviews and AI Mode use the normal Googlebot crawl. Use the Search generative AI control in Search Console instead, or snippet controls such as nosnippet. Google-Extended does not affect Google Search.

Do AI crawlers respect robots.txt?

The major search and training crawlers from OpenAI, Anthropic, Google, Apple, Perplexity, Amazon, Meta and Common Crawl say they follow robots.txt. Some user-triggered fetchers do not: Perplexity says Perplexity-User generally ignores it, and OpenAI says robots.txt rules may not apply to ChatGPT-User.

How long do robots.txt changes take to apply to AI crawlers?

OpenAI and Perplexity both say about 24 hours. DuckDuckGo says 72 hours for DuckAssistBot, and Amazon says about 24 hours for its agents. Google needs to recrawl the file and the affected pages, which can take longer.

Should I list AI bots by name if I allow everything?

Not necessarily. A single User-agent: * group with Allow: / covers every compliant crawler. Name a bot only when it needs different rules, and remember that a named group replaces the * group for that bot entirely.