How to check if AI bots can crawl my website: robots.txt, CDN rules and logs
A step-by-step way to check whether AI bots can crawl your website: robots.txt, firewall and CDN rules, a user-agent test, and logs that prove which bots arrive.
The short answer
Check four layers, in this order: your robots.txt file, your firewall or CDN rules, a direct request that imitates the bot, and your logs. A page can be allowed by robots.txt and still blocked at the edge, which is the most common surprise. Only logs show what bots actually experienced.
For ongoing monitoring, Promptwatch records real-time, IP-verified AI crawler logs and shows errors with explanations. For a quick first pass, you can do everything below with free tools.
Which bots to check
Test by name, because rules are written per user agent. The ones that matter:
- OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User
- Anthropic: ClaudeBot, Claude-User, Claude-SearchBot
- Perplexity: PerplexityBot, Perplexity-User
- Google: Google-Agent, GoogleOther
- Meta: Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher
- Mistral: MistralAI-User
Use the exact names above, because a rule written for an invented name does nothing. Different bots serve different purposes, such as training, search indexing or fetching a page when a user asks, and you may want to allow some and block others. Make that choice deliberately.
Step 1: read robots.txt
Open /robots.txt on your domain. Look for rules that name these bots, and for a wildcard User-agent: * group with Disallow: /, which blocks everything not matched by a more specific group. Remember that rules apply per user-agent group, and a more specific group usually wins over the wildcard. Google's Search Central documentation explains how groups are matched and how to test the file, and the same logic is a good guide for other crawlers.
Check each operator's documentation for how its user-triggered fetchers treat robots.txt, since some behave differently from background crawlers.
Mistakes we see often: a leftover Disallow: / from a staging site, a wildcard rule added for scrapers, and a typo in a bot name.
Step 2: check the firewall and CDN
Robots.txt is a polite request. A CDN or web application firewall can block bots outright, by user agent, IP range, rate or a "bot fight" setting. Look at your bot-management rules and any managed rules for AI crawlers.
If you use Cloudflare, its AI Crawl Control dashboard shows which AI crawlers request your pages, grouped by operator, with status codes, popular paths and robots.txt violations, and lets you allow or block each crawler. It is available on all plans, though on Free it identifies crawlers by user agent only, and referral metrics need a paid plan. Read more at Cloudflare AI Crawl Control.
Step 3: imitate the bot
A quick test from a terminal:
curl -I -A "GPTBot" example.com/your-page
Look at the status code. A 200 means that rule set let a request with that user agent through. A 403, 429 or a challenge page means something is blocking. Repeat with the other bot names.
Know the limits. This only tests rules based on the user-agent string. Real bots come from published IP ranges, and some firewalls trust or block by IP, so your request can pass or fail differently from a real crawler. Use it as a first screen, not proof.
Step 4: read the logs
Logs are the only proof of what happened. Look for requests from each bot name, the status codes returned, and the paths requested. Verify IP addresses, because anyone can fake a user-agent string.
Several tools do this for you.
| Tool | What it does | Starting price |
|---|---|---|
| Promptwatch | Real-time IP-verified crawler logs, errors with explanations, crawl-to-citation path, CSV export | Crawler logs from Professional $245/mo |
| Known Agents | Agent Analytics across many CDNs, spoof verification, robots.txt violations, Automatic Robots.txt | Free; Startup $29/mo |
| Cloudflare AI Crawl Control | Free dashboard if you are behind Cloudflare | Included on all plans |
| Scrunch AI | AI agent and bot traffic reporting | Core $250/mo |
| Trackbase | Crawler analytics via Cloudflare Worker, Logpush or WordPress plugin | Essential €49/mo |
What Promptwatch adds
Agent Analytics filters out spoofed user agents and tracks the bots above. It explains failures, supports stackable path filters, and exports CSV. Logs can come from Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP source or the WordPress plugin.
The part a log-only tool cannot show is the next step. Promptwatch joins the crawl to citations, giving a citation rate per page, so you can tell a page that is crawled and cited from one that is crawled and ignored. Crawler logs start on Professional at $245 a month, and Essential has none.
Promptwatch also offers free tools, including an Agentic Readiness Check, an AI Bots directory of more than 440 crawlers, and robots.txt and llms.txt generators, which help with the first pass.
Who should pick which
- Quick check, no budget: read robots.txt, run the curl test, and check your CDN dashboard.
- On Cloudflare: start with AI Crawl Control.
- You want bot visibility only, with a free plan: Known Agents.
- You want to know whether crawling leads to citations and traffic: Promptwatch.
What a check cannot tell you
Being able to crawl does not mean you will be cited. Access is a precondition, not a result. Once access is confirmed, the next questions are whether your pages answer real prompts and whether engines cite them, and no tool can promise that they will.