robots.txt Generator
Generate a robots.txt file with the right rules for search engines and AI crawlers. Copy-paste ready.
Search Engines
AI Crawlers
# robots.txt generated by AIExposureTool.com # Generated: 2026-04-15 # All other crawlers User-agent: * Disallow: /admin/ Disallow: /api/ Disallow: /private/
How to deploy
- Copy the robots.txt content above
- Save as
robots.txtin your site's root folder - Verify at
yoursite.com/robots.txt
Why this matters
Control AI access
Decide exactly which AI crawlers can read your site. Allow ChatGPT and Claude while blocking scrapers — or open everything for maximum visibility.
Avoid accidental blocks
Many default robots.txt files accidentally block AI crawlers. A single wrong rule can make your entire site invisible to ChatGPT and Perplexity.
Copy-paste ready
Generate a complete robots.txt with the right syntax. No guessing at crawler names or formatting. Just copy and deploy.
How it works
Select which crawlers to allow or block (Google, Bing, AI crawlers)
Add your sitemap URL and any custom paths to block
Copy the generated robots.txt and deploy to your site root
Frequently Asked Questions
What is a robots.txt file?
robots.txt is a text file at the root of your website (yoursite.com/robots.txt) that tells search engines and AI crawlers which pages they can and cannot access. It's one of the first files bots check when visiting your site.
Should I block AI crawlers in robots.txt?
It depends. If you want ChatGPT, Claude, and Perplexity to recommend your product, you should ALLOW AI crawlers. If you're a publisher worried about content scraping, you might block some. Our generator lets you choose per-crawler.
What AI crawlers should I know about?
The main AI crawlers are: GPTBot and ChatGPT-User (OpenAI), ClaudeBot and Claude-Web (Anthropic), PerplexityBot, Google-Extended (Gemini), Bytespider (ByteDance), CCBot (Common Crawl), and cohere-ai. Each can be individually allowed or blocked.
Does robots.txt affect my AI visibility?
Absolutely. If your robots.txt blocks GPTBot, ChatGPT literally cannot access your site to recommend it. Many sites accidentally block AI crawlers and wonder why they never appear in AI answers.
How do I add robots.txt to my site?
Create a file called robots.txt and upload it to your website's root directory so it's accessible at yoursite.com/robots.txt. Most hosting platforms (Vercel, Netlify, WordPress) have built-in ways to manage this file.
What's the difference between Disallow and Allow in robots.txt?
Disallow tells crawlers not to access a path. Allow explicitly permits access to a path (used to override a broader Disallow). For example, you might Disallow /admin/ but Allow /admin/public-page.
Related Free Tools
Robots.txt Analyzer
Analyze your existing robots.txt
AI Crawler Checker
Check which AI bots can reach your site
llms.txt Generator
Tell AI what your product does
Sitemap Checker
Check if your sitemap.xml exists
AI Visibility Audit
Full 25+ signal AI visibility scan
Meta Tag Checker
Check title and meta description
Want the complete picture?
Run a full AI visibility audit — 25+ signals, fix roadmap, and AI-generated files.