noindex

Two ways to set it

In the page’s HTML:

<meta name="robots" content="noindex">

Or as an HTTP response header, which also works for PDFs, images and other files that have no HTML head:

X-Robots-Tag: noindex

You can target one crawler by replacing robots with its name, for example googlebot.

The robots.txt trap

The most common noindex failure is blocking the same URL in robots.txt. Google says that if robots.txt stops it crawling a page, it never sees the noindex rule, and the URL can still appear in results if other pages link to it (Block indexing with noindex). For noindex to work, the page has to stay crawlable.

Other things to know

• It removes the page from AI Overviews and AI Mode too, since Google’s AI features only use indexed pages.
• It is not instant. The page drops out after Google recrawls it. For urgent cases, Search Console’s Removals tool hides a URL temporarily while the noindex takes effect.
• It does not save crawl budget. Google still has to fetch the page to read the rule.
• Do not combine it with a canonical tag pointing elsewhere; the two send conflicting instructions.

Our view: the costliest version of this mistake is a staging site’s site-wide noindex shipped to production during a launch. Check robots rules on your key templates after every release.

The meta tag checker reads both the meta robots tag and the X-Robots-Tag header for any URL.