OAI-SearchBot

The bot that controls ChatGPT search

Many site owners assume GPTBot decides whether ChatGPT can cite them. It does not. OpenAI’s crawler documentation says OAI-SearchBot is the agent used to surface websites in ChatGPT’s search features, and that sites opted out of it will not be shown in ChatGPT search answers, although they can still appear as plain navigational links.

Two further details from the same page:

• After you change robots.txt, OpenAI says it can take about 24 hours for its search systems to adjust.
• OpenAI publishes the IP ranges its crawlers use, so you can verify that a request claiming to be OAI-SearchBot really is.

Allow search, block training

If you want to be cited in ChatGPT but keep your content out of model training, give each agent its own group:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

A crawler follows the most specific group that names it, so OAI-SearchBot will not inherit rules written for * once it has its own group. Repeat any paths you want it to skip.

Look beyond robots.txt

robots.txt is only half the story. Firewalls, bot-management services and CDNs often block AI user agents with a 403 or a challenge page, regardless of what robots.txt allows. The free audit fetches your page as OAI-SearchBot and compares the response with a normal fetch, which catches this. The robots.txt tester checks the rules themselves, and ChatGPT SEO covers what happens after access is sorted.