GPTBot not blocked in robots.txt
OpenAI's crawler must be allowed to access your site if you want to appear in ChatGPT answers.
What is GPTBot?
GPTBot is OpenAI's web crawler, introduced in 2023. It is the bot that fetches pages to build the knowledge base ChatGPT uses when answering questions with web browsing enabled. If GPTBot cannot reach your site, you cannot be cited by ChatGPT.
Many sites accidentally blocked GPTBot during the 2023 panic about AI training. Some of those blocks are still in place years later, quietly cutting those sites out of ChatGPT results.
How to allow GPTBot
Check your robots.txt for any entry that disallows GPTBot. If you find one, remove it or explicitly allow the bot.
User-agent: GPTBot Allow: /
If you do want to keep GPTBot out of certain areas — for example, a login-walled app — you can allow the marketing site and disallow /app:
User-agent: GPTBot Allow: / Disallow: /app/