Homepage returns HTTP 200
Your homepage must respond with a 200 status code when an AI crawler fetches it.
Why this matters
AI crawlers make one request to your homepage and make a snap decision. If that request returns anything other than 200 — a redirect chain, a 403 from an over-aggressive firewall, a 500 from a flaky deploy — the crawler often gives up and moves on.
Unlike search engines, which retry and re-crawl for weeks, AI agents work in the context of a live conversation. They will not wait for your server to come back. A failed fetch means you are invisible for that conversation.
Common causes of non-200 responses
- — Redirect chains from http to https to www to non-www. Flatten these into a single 301.
- — Cloudflare or WAF rules blocking unfamiliar user agents. Whitelist GPTBot, ClaudeBot, and PerplexityBot.
- — Geoblocking. If you only serve EU traffic, you are invisible to crawlers based elsewhere.
- — Cold-start 500s on serverless deploys. Use a warming strategy or move to always-on hosting.
- — Authentication walls in front of the homepage. Your public site must be public.
How to check
Use curl with a realistic user agent and make sure the final response is 200.
curl -I -A "GPTBot" https://yourdomain.com # HTTP/2 200