GPTBot
Definition: GPTBot is OpenAI's web crawler for collecting content that may be used to train its generative AI models; blocking it opts your site out of training but does not remove it from ChatGPT search.
Also called: OpenAI GPTBot, GPTBot crawler
Updated
One company, three user agents
OpenAI runs three separate agents, and each answers a different question about your content (OpenAI’s crawler documentation):
| User agent | What it is for | robots.txt |
|---|---|---|
| GPTBot | Crawling content that may be used to train OpenAI’s foundation models | Honored |
| OAI-SearchBot | Finding pages to show in ChatGPT search | Honored |
| ChatGPT-User | Visiting a page because a user asked ChatGPT to | May not apply |
OpenAI’s documentation says disallowing GPTBot signals that a site’s content should not be used in training. It says nothing about search, because search is OAI-SearchBot’s job.
How to block it
User-agent: GPTBot
Disallow: /This stops future training crawls. It does not remove anything already collected, and it has no effect on ChatGPT search.
The common mistake
The usual error is the reverse: a site owner blocks “all the OpenAI bots” to stay out of training and silently removes the site from ChatGPT search as well. The same thing happens with CDN or firewall settings that block “AI bots” as a category. Our view: decide training and search separately. Many sites block GPTBot and allow OAI-SearchBot, which keeps them eligible for citations.
Check what your rules actually do with the robots.txt tester, and see robots.txt for AI bots for tested configurations.