Robots.txt and XML sitemaps are the foundation of technical SEO crawl control. Get them wrong and Google never finds your best pages — or wastes crawl budget on junk. This guide covers syntax, setup, and the mistakes I see on Nepali sites weekly.

→ Robots.txt (dictionary) · XML Sitemap (dictionary) · Technical SEO Audit · Google Search Console Guide

robots.txt vs noindex vs canonical — quick reference

Tool Controls Use when
robots.txt Disallow Crawling Block admin, search results, faceted URLs
noindex meta tag Indexing Deindex thank-you pages, thin duplicates
canonical tag Preferred URL Consolidate duplicate versions
XML sitemap Discovery Help Google find important URLs

Critical: Disallow in robots.txt does not deindex. Google may still index a URL if external sites link to it. Use noindex for deindexing.

robots.txt syntax

Location: https://yoursite.com/robots.txt (must return HTTP 200)

# Allow all crawlers, block admin and internal search
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

# Sitemap location (required)
Sitemap: https://arjankc.com.np/sitemap.xml

Rules

Directive Meaning
User-agent: * Applies to all bots
User-agent: Googlebot Google-specific rules
Disallow: /path/ Don’t crawl URLs starting with /path/
Allow: /path/file Exception within disallowed path
Sitemap: Points to XML sitemap URL

Never block

  • CSS and JavaScript files (/wp-content/themes/, /assets/js/)
  • Image directories needed for rendering
  • Your sitemap URL

Google needs CSS/JS to render pages correctly. Blocking them causes “page cannot be indexed” errors in Search Console.

Common robots.txt mistakes

Mistake Fix
Disallow: / on live site Remove — blocks entire site
Blocking /wp-content/ Allow — breaks rendering
Using Disallow to deindex Add noindex meta tag instead
No Sitemap directive Add Sitemap: https://...
Staging robots.txt on production Audit after every deploy

WordPress “Discourage search engines”

Settings → Reading → “Discourage search engines from indexing this site” adds:

User-agent: *
Disallow: /

Check this after every launch. It’s the #1 cause of zero organic traffic on new Nepali business sites.

XML sitemap structure

Standard sitemap

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://arjankc.com.np/blog/search-intent-seo-guide-2026/</loc>
    <lastmod>2026-08-11</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
    </url>
</urlset>

Limits

  • 50,000 URLs or 50 MB per sitemap file
  • Use sitemap index for larger sites:
<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://arjankc.com.np/post-sitemap.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://arjankc.com.np/page-sitemap.xml</loc>
  </sitemap>
</sitemapindex>

Include only

  • Indexable 200 OK URLs
  • Canonical versions (not duplicates)
  • Pages you want in search results

Exclude

  • noindexed pages
  • Redirect URLs (301/302)
  • 404 pages
  • Paginated archive pages (usually)
  • Admin, login, cart, checkout

WordPress setup

Plugin Sitemap robots.txt
Yoast SEO /sitemap_index.xml Virtual robots.txt (override in file)
Rank Math /sitemap_index.xml Settings → Edit robots.txt
All in One SEO /sitemap.xml Tools → robots.txt Editor

After install: submit sitemap in GSC → Sitemaps → Add new sitemap.

For static/Jekyll sites: generate sitemap at build time and place at /sitemap.xml.

Google Search Console submission

  1. Verify property (domain or URL prefix)
  2. Sitemaps → enter sitemap URL → Submit
  3. Monitor Pages report for indexing status
  4. Use URL Inspection → Request indexing for new priority pages

→ Google Search Console Guide

Crawl budget optimization

For sites with 1,000+ pages:

Action Why
Remove thin/duplicate URLs from sitemap Stop inviting crawl of junk
Fix redirect chains One hop saves crawl cycles
Block faceted search URLs in robots.txt Prevent infinite URL spaces
Strengthen internal links to key pages Discovery without sitemap dependency
Update lastmod only when content changes Signal freshness accurately

→ Crawl Budget (dictionary) · Index Bloat (dictionary)

Nepal-specific checklist

# Check
1 robots.txt returns 200 at domain root
2 “Discourage search engines” is OFF in WordPress
3 CSS/JS not blocked in robots.txt
4 Sitemap submitted in GSC
5 Sitemap excludes noindexed and redirect URLs
6 After domain migration — new sitemap submitted, old property monitored
7 Nepali + English URL variants have canonicals, not duplicate sitemap entries

Audit checklist

# Task Tool
1 robots.txt accessible and valid Browser, GSC robots.txt tester
2 No accidental site-wide Disallow Manual review
3 Sitemap valid XML, under limits XML validator
4 Sitemap URLs return 200 Screaming Frog, Sitebulb
5 Sitemap submitted in GSC Search Console
6 Coverage report — no unexpected “Blocked by robots.txt” GSC Pages
7 Internal links exist for key pages Site crawl

→ Technical SEO Audit (100 points)