Crawl budget
Definition: Crawl budget is the set of URLs a search engine can and wants to crawl on your site in a given period, determined by how much load your server can take and how much demand there is for your content.
Also called: crawl capacity, crawl rate, crawl demand
Updated
Do you need to worry about it?
Probably not, unless your site is large. Google says its crawl budget guidance is aimed at sites with roughly a million or more unique pages that change weekly, or ten thousand or more that change daily, and that these numbers are rough estimates (Google’s crawl budget guide). On smaller sites, pages that are not indexed are usually held back by quality or duplication, not by a crawl budget limit.
The two parts
Google describes crawl budget as the URLs it can and wants to crawl, made up of:
- Crawl capacity limit: how much crawling your server can handle without slowing down. Fast, error-free responses raise it; timeouts and 5xx errors lower it.
- Crawl demand: how much Google wants to crawl, based on the site’s size, how often it changes, page quality and relevance.
Where budget actually gets wasted
- Infinite URL spaces from faceted navigation, sort orders, session IDs and calendars. Block patterns you never want crawled in robots.txt.
- Redirect chains that cost several requests per URL. Point links straight at the final URL.
- Soft 404s and error pages returning 200.
- Sitemaps full of non-canonical, redirected or noindexed URLs.
Two things Google advises against: using noindex to save budget, since Google still has to request the page to see the rule, and blocking pages in robots.txt in the hope that Google will spend the freed budget elsewhere, which it does only if it was already hitting your capacity limit.
Check your sitemap’s hygiene with the sitemap checker and trace hops with the redirect checker.