ClaudeBot
Definition: ClaudeBot is Anthropic's web crawler for collecting public web content that may be used to train Claude models; Anthropic runs separate agents, Claude-SearchBot and Claude-User, for search and user requests.
Also called: Anthropic crawler, Claude-SearchBot, Claude-User
Updated
Anthropic’s three agents
Anthropic, like OpenAI, separates training from search and from user requests (Anthropic’s crawler policy):
| User agent | Role |
|---|---|
| ClaudeBot | Collects public web content that could contribute to model training |
| Claude-SearchBot | Crawls to improve the quality of Claude’s search results |
| Claude-User | Fetches pages when a Claude user asks a question that needs them |
Anthropic says all three honor robots.txt and that it supports the non-standard Crawl-delay directive, which Google ignores.
Blocking training only
To keep your content out of training while staying available to Claude’s search and to users who ask about you:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /One difference worth knowing
Claude-User respects robots.txt, which OpenAI’s ChatGPT-User and Perplexity’s Perplexity-User may not. So a broad rule such as User-agent: * with Disallow: /, or a rule that names Claude-User, does stop Claude from reading your page even when a person explicitly asks it to. That is the intended behavior, but it surprises site owners who blocked everything to opt out of training and then find Claude cannot answer questions about their own product.
Our view: decide training, search and user access as three separate questions, for each provider. The robots.txt tester shows which group each Anthropic agent actually matches.