Definition

Crawl budget is the number of pages search engine bots will crawl on your site within a given timeframe — determined by crawl capacity (server health) and crawl demand (URL importance and freshness).

Detailed Explanation

Google allocates limited crawl resources per site. Small sites (<1,000 pages) rarely hit limits. Large sites, e-commerce catalogs, and sites with many low-value URLs (filters, tags, pagination) must actively manage crawl budget.

Improve efficiency: fix 404s/500s, eliminate duplicate URLs via canonicals, block low-value paths in robots.txt, improve page speed, and use internal linking to prioritize important pages.

Nepal Context

Nepali news portals and large e-commerce catalogs (faceted filters, city pages) are most at risk. Block internal search results, thin tag pages, and parameterized URLs from crawling. Prioritize product, category, and service money pages.

Practical Examples

  1. Audit “Crawled - currently not indexed” in Search Console — often crawl budget waste.
  2. robots.txt Disallow: /search/, /*?sort=, /tag/ if thin.
  3. Log file analysis: which URLs consume most crawl slots?

Key Takeaways

  • Crawl budget matters for large/thin-content-heavy sites.
  • Server errors and slow TTFB reduce crawl capacity.
  • Quality internal links increase crawl demand for key pages.

Common Mistakes

  1. Blocking CSS/JS in robots.txt — hurts rendering and indexing.
  2. Infinite faceted navigation — millions of crawlable filter combinations.
  3. Ignoring Search Console crawl stats — miss server capacity issues.