Skip to content
SEOuchAudit my site
Menu

robots.txt

Definition: robots.txt is a plain-text file at the root of a host that tells crawlers which URL paths they may or may not request, using rules grouped by user agent; it controls crawling, not indexing.

Also called: robots exclusion protocol, robots.txt file

Updated

An example file

User-agent: *
Disallow: /cart/
Disallow: /search

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

This lets all crawlers in except for the cart and internal search pages, keeps OpenAI’s training crawler out entirely and points to the sitemap.

Rules that trip people up

  • It controls crawling, not indexing. Google says a URL blocked by robots.txt can still be indexed, without its content, if other pages link to it (robots.txt introduction). To keep a page out of results, allow crawling and use noindex.
  • Groups do not combine. A crawler obeys the most specific group that names it and ignores the * group. In the example above, if you later added Allow: / for GPTBot, it would no longer skip /cart/, because the * rules do not carry over.
  • It is per host and protocol. https://shop.example.com needs its own file.
  • The longest matching rule wins when Allow and Disallow overlap.
  • Google ignores Crawl-delay. Some other crawlers, including Anthropic’s, support it.

The format has been an internet standard since 2022, published as RFC 9309.

AI crawlers and robots.txt

Search and training crawlers from OpenAI, Anthropic, Google and Perplexity document that they follow robots.txt. User-triggered fetchers are different: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. Firewalls and CDNs can also block bots that robots.txt allows, so check both layers.

Test every rule against real URLs and user agents with the robots.txt tester before you deploy, and see robots.txt for AI bots for tested configurations.