robots.txt for AI Crawlers
If your robots.txt blocks GPTBot, ClaudeBot, or PerplexityBot, you can't be cited in AI answers — no matter how good your content is. Here's which AI crawlers to allow, a copy-paste robots.txt, and the mistakes that quietly make sites invisible to AI search.
On this page▾
1Why robots.txt Matters for AI
Your robots.txt file tells crawlers which parts of your site they may access — and if it blocks the AI crawlers, you disappear from AI answers entirely. This is the most common and most fixable reason brands never show up in ChatGPT, Perplexity, or Claude. Before any content strategy matters, the engines have to be allowed to read you.
It's the same gatekeeper for AI search that it's always been for Google — see does ChatGPT use my website for how access ties directly to citation eligibility.
2The AI Crawlers to Know
| Crawler | Owner | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data collection |
| OAI-SearchBot | OpenAI | ChatGPT search indexing |
| ChatGPT-User | OpenAI | Live page fetch for browsing |
| ClaudeBot | Anthropic | Claude web features |
| PerplexityBot | Perplexity | Perplexity search indexing |
| Google-Extended | Gemini / AI training control |
Free AI Crawler Checker
Paste your domain and see exactly which AI crawlers your current robots.txt allows or blocks.
3Training vs. Search Crawlers
This distinction trips up most people. Training crawlers (GPTBot, Google-Extended) gather data that may improve models. Search crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot) fetch pages to answer live queries with citations.
If your goal is AI-search visibility, the search crawlers are the ones you must allow. You can independently decide on training — some brands allow it for reach, others block it on principle. Blocking GPTBot does not remove you from ChatGPT search, because that runs on OAI-SearchBot and ChatGPT-User.
4Copy-Paste robots.txt
Here's a sensible starting point that welcomes AI search crawlers. Replace the sitemap URL with yours:
# Allow AI search crawlers, plus everyone else
User-agent: *
Allow: /
Disallow: /admin
Disallow: /dashboard/
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://example.com/sitemap.xmlNeed to build one from scratch or validate what you have? The robots.txt generator and robots.txt analyzer handle both, and our robots.txt for AI docs go deeper on directives.
5Mistakes to Avoid
- A blanket Disallow: / that accidentally blocks every crawler, including AI search bots.
- Blocking OAI-SearchBot or PerplexityBot while intending only to block training — check each user agent's actual purpose.
- Relying on robots.txt to hide private pages — it's a crawling directive, not access control. Use real auth for anything sensitive.
- Forgetting the Sitemap line, which helps every crawler discover your pages.
- Assuming robots.txt is enough — you still need server-rendered, citable content to be cited.
The subtle one is the third: robots.txt is a crawling directive, not access control. A disallowed page can still be discovered and is not protected — use real authentication for anything private. And remember, unlocking the door only makes you eligible; being cited still depends on citable content, which is where our how AI engines pick sources guide picks up.
Frequently Asked Questions
Should I allow AI crawlers in robots.txt?
If you want to appear in AI search answers, yes — allow the search-oriented crawlers like OAI-SearchBot, ChatGPT-User, PerplexityBot, and ClaudeBot. Blocking them removes you from consideration in ChatGPT, Perplexity, and Claude answers. You can separately decide whether to allow training crawlers, which is a distinct choice.
What's the difference between AI training and AI search crawlers?
Training crawlers (like GPTBot and Google-Extended) gather data that may be used to improve models. Search crawlers (like OAI-SearchBot and PerplexityBot) index and fetch pages to answer live queries with citations. If your goal is AI-search visibility, prioritize allowing the search crawlers; training access is optional and separate.
Does blocking GPTBot remove me from ChatGPT?
Not from ChatGPT search. GPTBot is primarily a training crawler; ChatGPT's web search uses OAI-SearchBot and ChatGPT-User. Blocking GPTBot only opts you out of training data. To stay visible in ChatGPT's answers, keep OAI-SearchBot and ChatGPT-User allowed.
Where should robots.txt live and what format does it use?
It must be at the root of your domain (example.com/robots.txt) and follow the standard User-agent / Allow / Disallow format. List each AI crawler as its own User-agent block with the paths it may or may not access, and include a Sitemap line. Test it after deploying to confirm it serves correctly.
Will robots.txt alone get me into AI answers?
No — it's necessary but not sufficient. Allowing crawlers only makes you eligible. You still need server-rendered, citable content and authority signals to actually be cited. Think of robots.txt as unlocking the door; the content is what gets you invited in.