robots.txt rules for AI crawlers: what to allow, what to block, how to test
Copy-ready robots.txt patterns for AI search visibility, the mistakes that quietly block citations, and how to test the result against real crawler logs.
Quick verdict
A robots.txt file is a request, not a lock, and it is read by each bot separately. For AI search visibility the safe pattern is: allow the search and user-triggered bots, make a deliberate choice about training bots, and test with real logs rather than assuming the file works. The testing step is where Promptwatch fits, since its Agent Analytics shows IP-verified bot visits and failure reasons instead of leaving you to guess.
How the file is actually read
A crawler looks for the most specific User-agent group that names it. If none matches, it falls back to the * group. That detail causes most accidents. A site adds User-agent: * with Disallow: /private/ and thinks it is done, while another line further down blocks everything for one named bot.
Remember three behaviors:
- Rules are per user agent, so naming GPTBot does not touch OAI-SearchBot.
- A longer, more specific path rule generally wins over a shorter one for the same agent.
- A file that returns a server error may be treated as fully restrictive by some crawlers, so keep it reachable.
Pattern A: open to visibility
For a brand that wants to be cited, allow the search and fetch agents explicitly:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
Explicit allow lines are not strictly required if nothing blocks these bots, but they document intent and survive a later wildcard disallow.
Pattern B: allow search, block training
If you object to training use but still want citations, add:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
The cost is a judgment call. You keep search visibility, and you opt out of bulk collection. Whether being absent from training data changes how often a model mentions you when it does not search is not something we can quantify, so do not let anyone sell you a number for it.
Pattern C: protect private areas only
Most sites do not need anything dramatic. Keep your existing disallows for admin paths, carts, and staging, and add nothing for AI bots. This is the default we would choose for a normal marketing site.
Mistakes that block citations quietly
| Mistake | What happens | Fix |
|---|---|---|
User-agent: * with Disallow: / left from staging | All bots blocked, including search bots | Remove after launch, add a launch checklist item |
| A rule for "ChatGPTBot" | Does nothing, the bot does not exist | Use GPTBot, OAI-SearchBot, ChatGPT-User |
Blocking /blog/ or /guides/ | Your most citable content is hidden | Narrow the disallow |
| Allowing in robots.txt, blocking in the firewall | Bot sees 403 | Allowlist verified ranges in the WAF |
| Migrating domains without moving the file | New host serves a different or missing file | Diff the files before cutover |
The firewall is the second gate
Your robots.txt can say yes while your CDN says no. Bot management tools often classify AI crawlers as automated traffic by default. The result looks like a robots success and a logs failure: bots request pages and receive 403 or 429.
This is why testing against logs matters. Promptwatch's Agent Analytics tracks GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Agent, GoogleOther, the Meta fetchers, and MistralAI-User. It reports errors with explanations, supports stackable path filters, and exports CSV. Logs arrive through Cloudflare, CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP endpoint, or the WordPress plugin.
A test routine
- Save the current file in version control before changing it.
- Make the change.
- Fetch
yourdomain.com/robots.txtin a browser and confirm the live content. - Wait for crawlers to return, typically days rather than minutes.
- Check logs for each bot you allowed. Expect 200 responses on public pages.
- Check logs for each bot you blocked. Expect no fetches of disallowed paths from well-behaved bots.
- Note any bot hitting paths you disallowed, since that bot is ignoring the file.
Promptwatch also offers free helpers: a robots.txt generator, an llms.txt generator, an AI Bots directory covering 440+ crawlers, and an Agentic Readiness Check. Use them to draft, then verify with logs.
Cost of verifying
Crawler logs start on Professional at $245 a month (25M logs). Essential does not include them. Business is $579 with 100M logs, and every agency plan has crawler logs from $199 a month. For a single site with one robots.txt change, your CDN's own log export is free and enough. For ongoing monitoring across many sites, a platform pays back the setup time.
Who should pick which
| You are... | Do this |
|---|---|
| Launching a new site | Add a launch checklist item to remove staging blocks |
| A publisher with training concerns | Pattern B, then verify with logs |
| A brand chasing citations | Pattern A, plus a firewall review |
| An agency auditing client sites | Agency plans with crawler logs, run the test routine per client |
| Not sure what is hitting you | Read your logs this week, or connect them to Promptwatch |
You can compare other platforms in our ranking. For this particular job, a tool without crawler logs cannot confirm your rules worked, so we would start with the one that has them.
A closing rule
Never treat robots.txt as proof of anything. It states intent. Logs state outcome. When the two disagree, believe the logs, fix the gate that is actually closed, and write down which gate it was so the next migration does not reopen the same problem.