robots.txt tester

A Disallow rule in robots.txt stops crawling, not indexing. Google says a blocked URL can still be indexed if other pages link to it, usually without a description (introduction to robots.txt). So the useful question is narrower: which crawler may fetch which URL, and which line decides it. This robots.txt tester answers that for Googlebot and the main AI crawlers, and works as a robots txt checker for the mistakes that quietly block the wrong things.

Enter a site URL or paste a draft, pick a user agent and the path to test. You get the verdict and the matching rule, the full file with that line highlighted, and the same path checked for every major AI crawler.

Why use a robots.txt tester now that Google’s is gone?

Google retired its robots.txt Tester in 2023 and replaced it with the robots.txt report in Search Console. The report lists the robots.txt files Google found for your verified sites, when it last fetched them and any parsing problems. It does not test a draft before you publish it, or check a path for AI crawlers, which is what this tool does.

How the verdict is decided

The tester follows Google’s published interpretation of robots.txt (How Google interprets the robots.txt specification), which is also standardised as RFC 9309. Matching uses the open-source robots-parser library, which mirrors the matching logic of Google’s own open-source parser.

1. Pick the group. A crawler obeys the group whose User-agent line names it. If none does, it uses the * group. Several groups for the same crawler are merged; a specific group and the * group are never combined.
2. Match the path. Allow and Disallow values match from the start of the path, with * as a wildcard and $ marking the end of the URL. Paths are case-sensitive.
3. The longest rule wins. When an Allow and a Disallow match equally, the less restrictive Allow wins.
4. No match means allowed.

The response code of the file matters as much as its content. Google says it treats 4xx responses other than 429 as if there were no robots.txt, so everything may be crawled. A 5xx or 429 response makes Google stop crawling the site for 12 hours, then use its last cached copy for up to 30 days. The tester shows which case applies before anything else.

AI crawlers: who the rules affect

AI companies run separate crawlers for separate jobs, and blocking one does not block the others. The bot table groups them by purpose:

• Search and answer crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes behind AI search answers. OpenAI says sites that block OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links.
• User-triggered fetchers such as ChatGPT-User, Claude-User and Perplexity-User load a page when someone asks about it. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt, because a person started the request.
• Training crawlers and tokens such as GPTBot, ClaudeBot, CCBot and Google-Extended. Google-Extended is a token, not a crawler: it controls whether content Google has crawled may be used for Gemini, and it does not affect Google Search, including AI Overviews, which rely on Googlebot.

OpenAI documents its crawlers on its bots page. For every crawler with suggested rules, see the AI crawlers guide and our robots.txt examples for AI bots.

Mistakes the tester looks for

• Disallow: / in the * group, which blocks the whole site for every crawler without a group of its own.
• Rules that block CSS, JavaScript or image folders such as /wp-content/, which Google needs to render pages.
• A missing Sitemap line, or one with a relative URL.
• Rules before the first User-agent line, which are ignored, and rules after a blank line, which older parsers may read as a new section.
• Misspelled fields such as Dissallow, and unsupported ones such as Noindex, which Google stopped honouring in 2019.
• Crawl-delay, which Google ignores, and files over 500 KiB, where Google ignores the rest.
• A byte order mark, invalid UTF-8, or an HTML page served at /robots.txt.

What robots.txt cannot do

robots.txt is a request that well-behaved crawlers honour, not access control. To keep a page out of Google, let it be crawled and use noindex: a page blocked in robots.txt cannot show Google its noindex tag. To stop a crawler that ignores the file, block it at your server or CDN. This tester reads only the file, so a firewall that challenges bots can still block a crawler that robots.txt allows. The free SEO audit fetches your page as several AI crawlers to catch that, and the sitemap checker confirms your Sitemap line points to a working file.