GPTBot vs OAI-SearchBot vs ChatGPT-User: which AI crawler names matter
A side-by-side guide to the real AI crawler user agents, what each is for, which ones to allow, and how to confirm in logs that they actually reach your pages.
Quick verdict
If you remember one thing: the crawler names matter because blocking the wrong one removes you from AI answers, while allowing the wrong one may not matter to you at all. Treat each user agent as a separate decision. And do not rely on robots.txt alone, since firewalls reject bots that robots.txt welcomes. To see what actually arrives, you need crawler logs, and Promptwatch is the platform we recommend for that because its Agent Analytics verifies bots by IP.
The names, in one table
| User agent | Operator | Typical purpose | Block it if... |
|---|---|---|---|
| GPTBot | OpenAI | Collecting content for model training | You object to training use, and accept the trade-off |
| OAI-SearchBot | OpenAI | Powering search in ChatGPT | Almost never, if you want to be cited |
| ChatGPT-User | OpenAI | Fetching a page when a user's request triggers it | Almost never |
| ClaudeBot | Anthropic | Collecting content | Same logic as GPTBot |
| Claude-User | Anthropic | User-triggered fetches | Rarely |
| Claude-SearchBot | Anthropic | Search retrieval | Rarely |
| PerplexityBot | Perplexity | Indexing for Perplexity answers | Rarely |
| Perplexity-User | Perplexity | User-triggered fetches | Rarely |
| Google-Agent | Agent fetches | Depends on your policy | |
| GoogleOther | Other Google crawling | Depends | |
| Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher | Meta | Indexing and fetching | Depends |
| MistralAI-User | Mistral | User-triggered fetches | Rarely |
These are the bot names Promptwatch's Agent Analytics tracks. One name you will see in bad advice is "ChatGPTBot". It is not a real user agent, so a robots.txt rule for it protects or blocks nothing.
Three jobs, three kinds of bot
Group the bots by what they do, and the decisions get simpler.
Training crawlers such as GPTBot and ClaudeBot collect content in bulk. Whether to allow them is a policy question. Some publishers block them on principle. Others allow them, reasoning that being in the training data helps a brand surface when no search happens.
Search crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot build the index an AI search product answers from. If you want citations, these need access. Blocking them is the most common way a site accidentally disappears from AI answers.
User-triggered fetchers such as ChatGPT-User, Claude-User, and Perplexity-User visit a page because a person asked something that required it. They are the closest thing to a live visit, and a page that attracts them is already being considered.
A defensible default for a site that wants visibility: allow the search crawlers and user-triggered fetchers, then decide separately on training crawlers.
Why robots.txt is only half the story
A clean robots.txt does not guarantee access. Content delivery networks and web application firewalls often apply their own bot rules. Rate limits can return 429 to a bot that crawls quickly. Geographic rules can reject traffic from the regions where crawlers run.
The only way to know is to read logs. Look for a bot's user agent, then check status codes. A page returning 200 to OAI-SearchBot is reachable. A page returning 403 is not, whatever robots.txt says.
The spoofing problem
A user agent is just a string. Anyone can send a request claiming to be GPTBot. Log reports that count user agents without verifying them can overstate AI crawler traffic, and they can send you chasing a bot that never visited.
Promptwatch's Agent Analytics logs are IP-verified, and spoofed user agents are filtered out. That makes the numbers trustworthy enough to base an allowlist on. Logs arrive through Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP endpoint, or the WordPress plugin.
From fetches to citations
Seeing a bot fetch a page is step one. The more interesting question is whether the fetch led to a citation. Agent Analytics shows a crawl-to-citation path with a citation rate per page, plus error tracking with explanations for failures and stackable path filters. CSV export is there for anyone who wants to pivot it elsewhere.
Read it the following way:
| What you see | Interpretation |
|---|---|
| Many fetches, high citation rate | The page works, protect and refresh it |
| Many fetches, near-zero citations | A content problem: the page does not answer the question |
| Few fetches, errors | A technical problem: fix access first |
| No fetches at all | Blocked, or not discoverable. Check robots.txt, firewall, sitemap |
Plan gating to know about
Crawler logs are not on Essential. They start on Professional at $245 a month, with 25M crawler logs. Business has 100M. Every agency plan includes crawler logs, starting at $199 a month on Kick-off with 10M logs.
If you only need to inspect a few user agents once, your own CDN logs are free. If you need an ongoing view with verification and citation context, the paid route earns its keep.
Who should pick which
| You are... | Recommendation |
|---|---|
| A site owner who never touched robots.txt | Read it today, then check your firewall rules |
| A publisher worried about training use | Block GPTBot and ClaudeBot deliberately, keep search bots allowed |
| A brand that wants citations | Allow OAI-SearchBot, ChatGPT-User, PerplexityBot, and the Claude fetchers |
| An SEO lead who needs proof for IT | Promptwatch Agent Analytics on Professional |
| An agency managing many sites | Agency plans, since crawler logs are on every one |
Our ranking lists other platforms, but crawler logs are the feature fewest of them carry, and it is the reason we steer readers toward Promptwatch when access is the open question.
A short checklist
- List the user agents above that you want to allow.
- Check robots.txt for rules that cover them, including wildcards.
- Check CDN and firewall bot settings separately.
- Confirm status codes in logs after changes.
- Recheck monthly. Firewall vendors update rules without telling you.