Definition

Robots.txt is a text file at your site root (/robots.txt) that tells search engine crawlers which URLs they may or may not request — a crawl directive, not an indexation guarantee.

Detailed Explanation

User-agent: *
Disallow: /admin/
Disallow: /search/
Allow: /blog/

Sitemap: https://example.com/sitemap.xml

Disallow prevents crawling; it does not always prevent indexing if URLs are linked externally. Use noindex meta tag to prevent indexing.

Never block CSS, JS, or image paths needed for rendering.

Nepal Context

Nepali WordPress sites often accidentally Disallow /wp-content/ breaking rendering in Search Console. Audit robots.txt after every migration or SEO plugin install.

Key Takeaways

  • robots.txt = crawl control, not index control
  • Always include Sitemap directive
  • Test with Google Search Console robots.txt tester

Common Mistakes

  1. Blocking entire site during development and forgetting to unblock
  2. Using robots.txt to hide pages you want deindexed (use noindex instead)
  3. Blocking resources Google needs to render JavaScript pages