Sitemap present and reachable

Why sitemaps still matter

Sitemaps are old technology, but AI crawlers still use them. An XML sitemap at /sitemap.xml tells a crawler exactly which URLs exist and when they were last updated. That saves crawl budget and makes sure nothing is missed.

For AI agents specifically, sitemaps are most valuable on sites where the main content lives several clicks deep — blog archives, documentation, product catalogs. Without a sitemap, a crawler may only see what the homepage links to.

Minimal sitemap example

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://yourdomain.com/</loc>
    <lastmod>2026-04-01</lastmod>
  </url>
  <url>
    <loc>https://yourdomain.com/pricing</loc>
    <lastmod>2026-04-01</lastmod>
  </url>
</urlset>

Best practices

• — Reference the sitemap from robots.txt with a Sitemap: line.
• — Include lastmod dates — they help crawlers prioritize.
• — Keep each sitemap under 50,000 URLs and 50 MB.
• — If you have more than that, use a sitemap index.
• — Generate it automatically. Most frameworks have a plugin.