Crawl budget

Crawl budget is not a quota you are granted and can spend. It is the outcome of two independent limits meeting.

Crawl rate limit is how fast the search engine is willing to request pages without harming the site. It rises when the server responds quickly and consistently, and falls when responses slow down or return 5xx errors. This is the engine protecting your server, not rationing your visibility.

Crawl demand is how much the engine wants your URLs. It rises with popularity and with how often content genuinely changes, and falls for URLs the engine has learned are duplicates, low value, or stale.

When it actually matters

For most sites it does not. Google has stated that sites with fewer than a few thousand URLs are generally crawled efficiently and do not need to think about crawl budget at all. It becomes a real constraint on large sites, on sites that generate many URLs from faceted navigation or parameters, and on sites where a significant share of crawling is spent on URLs that should not exist.

What wastes it

• Faceted navigation and sort parameters producing near-infinite URL variations
• Session IDs or tracking parameters in internal links
• Soft 404s — pages returning 200 with an error message in the body
• Long redirect chains, where each hop is a separate fetch
• Duplicate content reachable at several URLs with no canonical signal

How to see it

Server access logs are the primary evidence: they show exactly which URLs a crawler requested, how often, and what status each request returned. The Crawl Stats report in Google Search Console shows the same activity from Google’s side, broken down by response code, file type and purpose. If a large share of requests is landing on URLs you did not intend to publish, that is crawl budget being consumed by something other than your content.