Crawl Budget.

Crawl budget is the number of pages a search engine will crawl on a website within a given time period. It's determined by two factors: crawl rate (how fast the server can be crawled without degrading performance—Google adjusts this based on server response) and crawl demand (how much Googlebot wants to crawl, based on popularity and freshness signals). Google allocates limited crawling capacity, so sites must ensure their budget is spent on pages worth indexing.

When Crawl Budget Matters

Crawl budget is primarily relevant for large sites (100k+ URLs). For most small-to-medium sites, Googlebot can crawl all important pages without difficulty. Large e-commerce sites, news publishers, and sites with faceted navigation are most affected—unchecked URL proliferation can cause important pages to be crawled infrequently or not at all.

Optimising Crawl Budget

Improve crawl budget efficiency by: blocking low-value URLs in robots.txt (faceted navigation, session ID parameters, admin pages); using canonical tags to consolidate duplicates; redirecting outdated URLs instead of leaving 404 errors; improving server speed to allow faster crawling; keeping the XML sitemap limited to indexable, high-value URLs; and reducing redirect chains that consume extra crawl hops.