AI crawlers list: GPTBot, OAI-SearchBot, ClaudeBot and more

AI crawlers in one reference table: operator, user agent, purpose, robots.txt behaviour, verification, what blocking each costs, and robots.txt policies for three goals.

Every AI crawler falls into one of three groups, and most robots.txt mistakes come from treating them as one. A training crawler collects pages to build future models. A search crawler builds the index that an assistant retrieves from when it answers. A user-triggered fetcher opens one page because a person asked an assistant about it. Each operator runs at least two of the three under different names, each name has its own rules, and blocking the wrong one costs you citations while blocking the right one costs you nothing. This reference lists every major AI crawler from its operator’s own documentation, explains what blocking each costs, gives copy-paste robots.txt policies for three goals, and shows how to tell a real crawler from a spoof.

The three kinds of AI bot

Kind	What it does	Honours robots.txt?	Example tokens
Training crawler	Crawls broadly to collect content for training or fine-tuning models	Yes, by documentation, for every operator listed here	`GPTBot`, `ClaudeBot`, `Google-Extended` (token), `Applebot-Extended` (token), `CCBot`, `meta-externalagent`, `MistralAI-Training`
Search crawler	Crawls to build a search index that an assistant cites from	Yes, by documentation	`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Googlebot`, `bingbot`, `Applebot`, `Meta-WebIndexer`, `Amzn-SearchBot`, `MistralAI-Index`, `DuckAssistBot`
User-triggered fetcher	Fetches a single page because a user asked	Often not; operators say the user’s request takes precedence	`ChatGPT-User`, `Claude-User`, `Perplexity-User`, `Meta-ExternalFetcher`, `Amzn-User`, `MistralAI-User`

The distinction that matters most: OpenAI says sites that opt out of OAI-SearchBot “will not be shown in ChatGPT search answers”, while disallowing GPTBot “indicates a site’s content should not be used in training”. The settings are independent. The same split applies at Anthropic (Claude-SearchBot against ClaudeBot), Perplexity (PerplexityBot is search-only and not used for training), Mistral (MistralAI-Index against MistralAI-Training) and Meta (Meta-WebIndexer against meta-externalagent).

The AI crawler reference table

Everything below comes from the operator’s own documentation, linked in the last column. Verification files are the JSON IP-range lists each operator publishes; “rDNS” means the operator also documents a reverse-DNS check.

Operator	User agent token	Kind	Honours robots.txt?	Verification	Documentation
OpenAI	`GPTBot`	Training	Yes	`openai.com/gptbot.json`	OpenAI crawlers
OpenAI	`OAI-SearchBot`	Search (ChatGPT search)	Yes	`openai.com/searchbot.json`	same
OpenAI	`ChatGPT-User`	User-triggered	“robots.txt rules may not apply”	`openai.com/chatgpt-user.json`	same
OpenAI	`OAI-AdsBot`	Ads landing-page checks	Not stated	`openai.com/adsbot.json`	same
Anthropic	`ClaudeBot`	Training	Yes	`claude.com/crawling/bots.json`	Anthropic crawlers
Anthropic	`Claude-SearchBot`	Search (Claude search)	Yes	same file	same
Anthropic	`Claude-User`	User-triggered	Yes	same file	same
Perplexity	`PerplexityBot`	Search (Perplexity)	Perplexity recommends allowing it; not used for training	`perplexity.ai/perplexitybot.json`	Perplexity crawlers
Perplexity	`Perplexity-User`	User-triggered	“generally ignores robots.txt rules”	`perplexity.ai/perplexity-user.json`	same
Google	`Googlebot`	Search, AI Overviews, AI Mode	Yes	`common-crawlers.json` and rDNS	Google crawlers
Google	`Google-Extended`	Training and grounding control token	Yes (a token, no crawler)	none; it never fetches	same
Google	`Google-CloudVertexBot`	Site-owner-requested Vertex AI crawl	Yes	Google range files	same
Microsoft	`bingbot`	Search (Bing, Copilot)	Yes	`bing.com/toolbox/bingbot.json`	Bing crawlers
Apple	`Applebot`	Search (Siri, Spotlight, Safari) and context for AI answers; data may also train models	Yes; follows `Googlebot` rules if none for Applebot	`applebot.json` and rDNS	Applebot
Apple	`Applebot-Extended`	Training control token	Yes (a token; “does not crawl webpages”)	none	same
Meta	`meta-externalagent`	Training and product indexing	Yes	RADb routes for AS32934	Meta crawlers
Meta	`Meta-WebIndexer`	Search (Meta AI citations)	Yes	same	same
Meta	`Meta-ExternalFetcher`	User-triggered, including agents	“may bypass robots.txt rules”	same	same
Meta	`facebookexternalhit`	Link previews	May bypass for integrity checks	same	same
Amazon	`Amazonbot`	Products and services; “may be used to train Amazon AI models”	Yes; no `Crawl-delay`	IP list page	Amazonbot
Amazon	`Amzn-SearchBot`	Search (Alexa); no generative training	Yes	same	same
Amazon	`Amzn-User`	User-triggered	“may not follow all robots.txt directives”	same	same
Common Crawl	`CCBot`	Open crawl archive used in training datasets	Yes	`index.commoncrawl.org/ccbot.json` and rDNS	CCBot
Mistral	`MistralAI-Training`	Training	Yes	none published	Mistral crawlers
Mistral	`MistralAI-Index`	Search (Mistral search)	Yes	`mistral.ai/mistralai-index-ips.json`	same
Mistral	`MistralAI-User`	User-triggered	Controls which sites user requests can reach	`mistral.ai/mistralai-user-ips.json`	same
DuckDuckGo	`DuckAssistBot`	AI-assisted answers; not used for training	Yes; opt-out takes 72 hours	`duckduckgo.com/duckassistbot.json`	DuckAssistBot
DuckDuckGo	`DuckDuckBot`	Search	Yes	`duckduckgo.com/duckduckbot.json`	same

The GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended glossary entries each hold the one-paragraph version for a single token.

Operator notes

The table hides nuance that changes decisions. These notes carry the operators’ own wording where it matters.

OpenAI

OpenAI documents four agents. OAI-SearchBot “is used to surface websites in search results in ChatGPT’s search features”; opting out removes a site from ChatGPT search answers, though it may still appear as a navigational link, and OpenAI says search opt-out changes take about 24 hours. GPTBot collects content that may train foundation models; “a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot”. If both are allowed, OpenAI may use one crawl for both purposes. ChatGPT-User “is not used for crawling the web in an automatic fashion” and, because its fetches are user-initiated, “robots.txt rules may not apply”. OpenAI recommends verifying by IP range and publishes a JSON file per agent.

Anthropic

Anthropic documents three agents and states that all of them “respect ‘do not crawl’ signals by honoring industry standard directives in robots.txt”. ClaudeBot collects content that could contribute to training; blocking it excludes future material from training datasets. Claude-SearchBot indexes content for search; Anthropic says blocking it “may reduce your site’s visibility and accuracy in user search results”. Claude-User fetches when a Claude user asks; blocking it “may reduce your site’s visibility for user-directed web search”. Anthropic supports the non-standard Crawl-delay directive, publishes a single IP list for all three agents, and advises against IP blocking because it “may not work correctly or persistently guarantee an opt-out”.

Perplexity

Perplexity documents two agents. PerplexityBot is “designed to surface and link websites in search results on Perplexity” and is not used to collect training data; Perplexity recommends allowing it. Perplexity-User fetches on a user’s behalf and “generally ignores robots.txt rules” because a person requested the page; Perplexity notes that web application firewalls may need to allow its bots explicitly. Separately, in August 2025 Cloudflare published a report accusing Perplexity of using undeclared crawlers with generic browser user agents and rotating networks to evade robots.txt and network blocks; Perplexity disputed the findings. Treat that as a contested vendor claim, and as a reason to verify by IP rather than by user-agent string.

Google

Google’s AI features need no special crawler. Google says a page must be indexed and eligible to be shown with a snippet to appear in AI Overviews and AI Mode, and that nosnippet and max-snippet also limit what can be used as a direct input to those features. Google-Extended “is a standalone product token” with no user agent of its own; it controls whether crawled content may train future Gemini models or be used for grounding, and Google says it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”. Blocking Googlebot removes you from Google entirely, AI features included. Google publishes range files for common crawlers, special-case crawlers and user-triggered fetchers, and documents a reverse-DNS check against googlebot.com, google.com and googleusercontent.com (Google’s verification guide).

Microsoft

Bing has no separate AI training crawler; bingbot feeds both Bing and Copilot. Control is through meta tags rather than user agents. Bing says content tagged noarchive “will not be included in Bing Chat answers” and will not train Microsoft’s generative models, content tagged nocache may be included but only as URL, title and snippet, a page carrying both is treated as nocache, and pages with either tag still appear in normal results (Bing’s 2023 announcement). Microsoft publishes bingbot’s IP ranges as JSON.

Apple

Applebot crawls for Spotlight, Siri and Safari, and Apple says the data may also help train its foundation models and supply context for AI-generated answers that link to sources. Applebot-Extended is a robots.txt token that controls whether already-crawled data may train Apple’s general-purpose models; Apple says it “does not crawl webpages” and that disallowing it does not remove pages from search. Applebot follows Googlebot rules when it finds none addressed to itself, ignores Crawl-delay, and supports robots meta tags. Verification is by reverse DNS to *.applebot.apple.com or the published range file.

Amazon

Amazonbot improves Amazon’s products and services and “may be used to train Amazon AI models”; it honours robots.txt and robots meta tags, treats noarchive as “no model training”, does not support Crawl-delay, and caches robots.txt for up to 30 days. Amzn-SearchBot supports Alexa search without generative training. Amzn-User fetches live information on request and may not follow every directive. Amazon publishes an IP list on its developer site.

Common Crawl

CCBot builds the open Common Crawl archive. It is not an AI company’s crawler, but the archive is the raw material for much of the industry: Research shows (a Mozilla Foundation report, February 2024) that at least 64 percent of 47 language models published between 2019 and October 2023 were trained on Common Crawl data, and that over 80 percent of GPT-3’s tokens came from it. Blocking CCBot therefore affects many future models at once, but has no effect on any live search product. Common Crawl publishes IP ranges and supports reverse DNS to *.crawl.commoncrawl.org for IPv4, and warns that other crawlers falsely identify as CCBot.

Mistral and DuckDuckGo

Mistral splits its traffic three ways: MistralAI-Training for datasets, MistralAI-Index for its search index, and MistralAI-User for user requests in its assistant; it says indexed content is not used for training. DuckDuckGo’s DuckAssistBot crawls in real time for AI-assisted answers that cite sources, is not used for training, and DuckDuckGo says opting out “doesn’t affect organic rankings” and takes effect after 72 hours.

What does blocking each crawler cost you?

If you disallow	You lose	You keep	Source
`GPTBot`	Your content in future OpenAI training data	ChatGPT search citations, if `OAI-SearchBot` is allowed	OpenAI
`OAI-SearchBot`	Appearance in ChatGPT search answers (navigational links may remain)	Nothing else changes	OpenAI
`ChatGPT-User`	Nothing reliable; OpenAI says the rule may not apply	A firewall rule is the only effective control	OpenAI
`ClaudeBot`	Your content in future Anthropic training data	Claude search and user fetches	Anthropic
`Claude-SearchBot`	Visibility and accuracy in Claude’s search results	Training opt-in or opt-out unaffected	Anthropic
`Claude-User`	Claude’s ability to open your page when a user asks	Search index inclusion	Anthropic
`PerplexityBot`	Appearance in Perplexity search results	Nothing else; it is not a training crawler	Perplexity
`Perplexity-User`	Nothing; it generally ignores robots.txt	A firewall rule is the only control	Perplexity
`Google-Extended`	Use of your content for Gemini training and grounding	Google Search, AI Overviews and AI Mode inclusion	Google
`Googlebot`	Google Search, Discover, AI Overviews and AI Mode	Nothing; this is total removal	Google
`bingbot`	Bing, Copilot and everything that licenses Bing’s index	Nothing; use `noarchive` instead to stay in results but out of answers	Microsoft
`Applebot`	Siri, Spotlight and Safari suggestions	Use `Applebot-Extended` instead to opt out of training only	Apple
`Applebot-Extended`	Use of your content for Apple model training	Apple search features	Apple
`meta-externalagent`	Meta training and product indexing	Link previews; Meta AI search if `Meta-WebIndexer` is allowed	Meta
`Amazonbot`	Alexa answers and possible Amazon training	Use `noarchive` to allow crawling but refuse training	Amazon
`CCBot`	Inclusion in the Common Crawl archive, and so in the many models trained from it	Every live search product	Common Crawl, Mozilla
`MistralAI-Training`	Your content in Mistral training data	Mistral search and user fetches	Mistral
`DuckAssistBot`	Citations in DuckDuckGo AI answers	Organic DuckDuckGo rankings	DuckDuckGo

Our view: blocking training crawlers is a legitimate policy choice with a cost that lands years away. Blocking search crawlers is a visibility decision with a cost that lands today. Blocking user-triggered fetchers is mostly symbolic. Most businesses should allow every search and user agent, decide on training per operator, and verify that their firewall agrees with their robots.txt.

Recommended robots.txt policies

One rule governs how these files are read. Google says a crawler uses “the group with the most specific user agent that matches the crawler’s user agent” and that “user agent specific groups and global groups (*) are not combined” (Google’s robots.txt specification). OpenAI, Anthropic and the others follow the same standard. The consequence is that a User-agent: GPTBot group must contain every rule you want GPTBot to obey; it inherits nothing from *. The snippets below are written with that in mind. Test every one with the robots.txt tester before deploying.

Goal 1: maximise citations (allow everything)

The default, written out so the intent is visible to the next person who edits the file. No AI-specific groups are needed, because every crawler falls back to *.

# Allow all search engines, AI search crawlers, user-triggered fetchers
# and training crawlers. Keep private paths out of all of them.
User-agent: *
Disallow: /admin/
Disallow: /api/
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Goal 2: allow search and user fetches, block training

Each training token gets its own group. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, bingbot, Applebot, Meta-WebIndexer, Amzn-SearchBot, MistralAI-Index, DuckAssistBot) and user fetchers have no group, so they use *. Amazonbot is listed because Amazon says it may train models; omit that group if Alexa answers matter to you.

User-agent: *
Disallow: /admin/
Disallow: /api/
Allow: /

# Training crawlers and training control tokens
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: MistralAI-Training
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

For Bing, add <meta name="bingbot" content="noarchive"> to pages you want out of Copilot answers and Microsoft training while staying in Bing results; robots.txt cannot express that split. For Google, remember that Google-Extended only affects Gemini training and grounding; there is no robots.txt token that removes you from AI Overviews while keeping you in Search.

Goal 3: block all AI use, keep classic search

This keeps Googlebot and bingbot for ordinary results and blocks every AI-specific token. Three limits apply. Google’s AI Overviews and AI Mode still draw on Googlebot’s index, so excluding a page from them requires nosnippet, max-snippet:0 or noindex on the page, each of which also changes how it appears in classic results. Bing’s Copilot needs the noarchive meta tag. And Perplexity-User, ChatGPT-User, Meta-ExternalFetcher and Amzn-User do not reliably obey this file, so blocking them needs a firewall rule keyed on verified IP ranges.

User-agent: *
Disallow: /admin/
Disallow: /api/
Allow: /

# AI search crawlers
User-agent: OAI-SearchBot
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Meta-WebIndexer
Disallow: /

User-agent: MistralAI-Index
Disallow: /

User-agent: DuckAssistBot
Disallow: /

# User-triggered fetchers (advisory; enforce at the firewall)
User-agent: ChatGPT-User
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: Meta-ExternalFetcher
Disallow: /

User-agent: Amzn-User
Disallow: /

User-agent: MistralAI-User
Disallow: /

# Training crawlers and control tokens
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: MistralAI-Training
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Our audit reads your robots.txt against all of these tokens and reports them in three groups (search, user-triggered, training), marks a blocked search crawler as an error and a blocked training crawler as a policy choice. The robots.txt examples for AI bots post has variants for WordPress, Shopify and Cloudflare-managed files, and the robots.txt glossary entry covers the syntax.

Content Signals and IndexNow: what they add

Two newer mechanisms get mentioned alongside robots.txt. Neither replaces it, and the honest status of each is “low cost, low certainty”.

Content Signals Policy. Cloudflare published it in September 2025 as an addition to robots.txt that states how content may be used after it is crawled, since standard rules only control access. The syntax is one line with three optional signals:

User-agent: *
Content-Signal: search=yes, ai-input=no, ai-train=no
Allow: /

search covers building an index and showing links and short excerpts, and explicitly excludes AI-generated summaries; ai-input covers feeding content into a model at answer time, such as grounding or retrieval-augmented generation; ai-train covers training and fine-tuning. A signal left out expresses no preference. Cloudflare says the signals “express preferences” rather than enforce anything, that “some companies might simply ignore them”, and it names no AI company that has committed to honour them; it recommends combining them with firewall and bot-management rules (Cloudflare’s Content Signals announcement). Cloudflare adds search=yes, ai-train=no to the managed robots.txt of customers who opted into blocking AI training, which it said covered over 3.8 million domains at launch. Our audit reports a Content-Signal line as informational and notes that no crawler has confirmed honouring it. Adding one is reasonable as a statement of intent; expecting it to change crawler behaviour is not yet justified.

IndexNow. A protocol for telling search engines that a URL was added, changed or removed, so they can consider recrawling it. The participating endpoints listed in its documentation are Bing, Naver, Seznam, Yandex, Yep and Amazon; Google is not among them, and a submission to one participant is shared with the rest. It works by hosting a key file at your root and sending URLs to the API, up to 10,000 per POST (IndexNow documentation). Its relevance to AI is indirect: Bing’s index feeds Copilot and is licensed to other assistants, so faster Bing indexing can mean faster appearance in those answers. It does nothing for Google, ChatGPT, Claude or Perplexity directly. Our audit reports an IndexNow key as informational.

How do you verify a real crawler against a spoof?

Never trust the user-agent string; anyone can send one. Verify by comparing the request’s IP address with the operator’s published range file, or, where the operator documents it, by reverse DNS followed by a forward lookup. Every operator in the table above except Mistral’s training crawler publishes ranges, and Google, Apple and Common Crawl also document reverse DNS.

The procedure, using Google’s documented method as the pattern:

Take the IP from your logs, not the user-agent string.
Run a reverse DNS lookup. For Google, the hostname must end in googlebot.com, google.com or googleusercontent.com; for Apple, applebot.apple.com; for Common Crawl, crawl.commoncrawl.org.
Run a forward lookup on that hostname and confirm it resolves back to the original IP. A spoofer can set any reverse record for an IP it controls; it cannot make the operator’s forward record point at it.
Where no reverse DNS is documented, match the IP against the JSON file. OpenAI, Anthropic, Perplexity, Apple, Common Crawl, DuckDuckGo, Amazon, Mistral, Google and Microsoft each publish one; the prefixes array lists CIDR ranges and a creationTime tells you how fresh it is. Fetch the files on a schedule, because the ranges change.
Block or rate-limit what fails. A request claiming to be GPTBot from a residential IP range is a scraper. Common Crawl explicitly warns that other crawlers falsely identify as CCBot.

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Two practical notes. CDN bot-management products maintain verified-bot lists built from these same files, which is why a WAF rule that blocks by user-agent string alone both lets spoofers through and risks blocking legitimate crawlers whose string changed. And our audit’s “Fetch as AI crawler” check is deliberately labelled a heuristic: it sends each crawler’s user-agent string from our own servers, so a 403 tells you a rule keyed on the string exists, not that a verified crawler would be refused.

What to do next

Decide your policy per crawler using the cost table, write it with the three-goal snippets as a starting point, and test every token with the robots.txt tester. Then run the AI visibility checker, which shows the crawler table for a URL and whether each agent is allowed, and the free audit at the top of this site, which adds the firewall heuristic, snippet controls and whether your content is readable without JavaScript. If the goal is citations rather than exclusion, the AI visibility guide explains what each platform’s documentation says it retrieves, and the llms.txt guide explains why that file is not the lever people hope it is.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?: No. OpenAI says GPTBot controls whether your content may be used to train its models, while OAI-SearchBot controls whether your site appears in ChatGPT search answers, and the two settings are independent. A site can disallow GPTBot and still be cited by ChatGPT search, provided OAI-SearchBot is allowed.
Is Google-Extended a crawler?: No. Google says Google-Extended is a standalone product token used in robots.txt to control whether content Google has already crawled may be used to train Gemini models or for grounding. It has no user agent of its own, and Google says it does not affect a site's inclusion in Google Search or act as a ranking signal. AI Overviews and AI Mode use ordinary Googlebot.
Which AI crawlers ignore robots.txt?: By their operators' own documentation: Perplexity-User generally ignores robots.txt because a person requested the fetch, OpenAI says robots.txt rules may not apply to ChatGPT-User for the same reason, Meta says Meta-ExternalFetcher may bypass robots.txt, and Amazon says Amzn-User may not follow all directives. Automatic crawlers from OpenAI, Anthropic, Google, Apple, Common Crawl and DuckDuckGo document honouring it.
How do I block all AI crawlers?: Add a Disallow group for each training and search token you want to exclude, since robots.txt groups are not merged and a crawler only reads the group that names it. Then add firewall rules for the user-triggered fetchers that do not honour robots.txt. You cannot keep a page out of Google's AI Overviews by robots.txt alone without also keeping it out of Google Search; use nosnippet or noindex for that.
How can I tell if a crawler is really from OpenAI or Google?: Do not trust the user-agent string. Compare the request's IP address with the operator's published range file (OpenAI, Anthropic, Google, Apple, Perplexity, Common Crawl, DuckDuckGo, Amazon and Mistral all publish one), or run a reverse DNS lookup followed by a forward lookup for operators that support it, such as Google, Apple and Common Crawl.
Should I use Cloudflare's Content Signals or IndexNow?: Content Signals is a voluntary robots.txt extension that states how content may be used after crawling; Cloudflare's own announcement says some companies may ignore it and names no AI company that has committed to honour it. IndexNow is a URL-submission protocol for Bing, Yandex, Naver, Seznam, Yep and Amazon, not Google. Both are low-cost and low-certainty; neither replaces robots.txt and firewall rules.