AI Crawler Checker
Is your robots.txt blocking ChatGPT, Claude, Perplexity, or Gemini from crawling your site?
Free. No signup required.
Want the full picture?
Run a complete AI visibility audit — 25+ signals, fix roadmap, and generated files.
Why this matters
40% of sites block AI crawlers
Most blocks are accidental — legacy robots.txt rules that were written before GPTBot existed. One line can make you invisible to ChatGPT.
No crawl = no recommendation
If ChatGPT can't read your site, it can't recommend you. It's that simple. The first step to AI visibility is making sure the door is open.
5-minute fix, permanent impact
Adding Allow rules for AI bots takes 5 minutes. The impact is permanent — every AI query about your category can now find your content.
How it works
Enter your website URL
We fetch your robots.txt and check all 16 AI crawlers
See which bots are blocked with copy-paste fix code
Frequently Asked Questions
How do I check if my robots.txt is blocking AI crawlers?
Enter your URL in our AI Crawler Checker above. It fetches your robots.txt and checks all 16 major AI crawlers including GPTBot (ChatGPT), ClaudeBot (Claude), PerplexityBot, Google-Extended (Gemini), and Meta-ExternalAgent. You'll see which are blocked and which are allowed.
What AI crawlers should I allow in robots.txt?
At minimum, allow GPTBot (ChatGPT), ClaudeBot (Claude), PerplexityBot (Perplexity), and Google-Extended (Gemini). These are the four major AI platforms where buyers search for products. Blocking them means AI can't read your site and won't recommend you.
Does blocking GPTBot affect my Google rankings?
No — GPTBot is separate from Googlebot. Blocking GPTBot only prevents ChatGPT from crawling your site. However, blocking Google-Extended prevents Gemini from using your content, which may affect AI Overviews visibility.
What is GPTBot and should I block it?
GPTBot is OpenAI's web crawler for ChatGPT. If you block it, ChatGPT can't access your site content and won't recommend your product in search results. Unless you have a specific reason to block it (like copyrighted content), you should allow GPTBot.
Over 40% of websites block AI crawlers — why?
Most blocks are accidental. Many websites have broad 'Disallow: /' rules or copy robots.txt templates that block all non-Google bots. AI crawlers like GPTBot didn't exist when most robots.txt files were written, so they get caught by legacy rules.
How do I fix a blocked AI crawler?
Add explicit Allow rules for each AI bot in your robots.txt. For example: 'User-agent: GPTBot\nAllow: /' for ChatGPT. Our checker shows the exact fix code for any blocked crawlers it detects.
Related Free Tools
Robots.txt Analyzer
Full robots.txt syntax analysis + AI grade
llms.txt Generator
Check if AI knows what your product does
AI Visibility Audit
Full 25+ signal AI visibility check
Security Headers
Check CSP, HSTS, and security headers
Sitemap Checker
Check if AI can discover your pages
Meta Tag Checker
Check what search engines see
Want the complete picture?
Run a full AI visibility audit — 25+ signals, fix roadmap, and AI-generated files.