AI crawls vs citation crawls: how tools count bot traffic
Two crawler logs can disagree about the same traffic because they bucket bots differently. Here is the split Promptwatch uses and the questions to put to any vendor.
Quick verdict
When two tools report different AI crawler numbers for the same site, the logs are rarely the problem. The buckets are. One tool may count every AI bot visit together, another may count only the visits that happen while a user is waiting for an answer. Neither is wrong, but you cannot compare them until you know the rules.
Promptwatch published its rule on November 4, 2026, and it is clear enough to use as a template. For other vendors, we only claim what their catalog entries here support, which is nothing about bucketing, so the rest of this post is a list of questions to ask them.
The Promptwatch classification
In its GEO 360 views, Promptwatch separates two ideas:
- AI crawls include search indexers such as OAI-SearchBot, Claude-SearchBot, and Meta-WebIndexer.
- Citation crawls are only bots that fetch a page while answering a user, like ChatGPT-User.
The practical effect is spelled out: citation crawl totals fall by as much as AI crawl totals rise, and page categories can shift. So if a chart looked different on November 4 than the day before, a reclassification is a candidate explanation before you suspect your site.
Comparison table
| Question | Promptwatch | Other tools |
|---|---|---|
| Are search indexers counted as AI crawls? | Yes (OAI-SearchBot, Claude-SearchBot, Meta-WebIndexer) | Ask the vendor |
| What counts as a citation crawl? | Bots that fetch a page while answering a user, such as ChatGPT-User | Ask the vendor |
| Are spoofed user agents removed? | Yes, logs are IP-verified against each operator's published ranges | Ask the vendor |
| Is there a citation rate per page? | Yes, in the crawl-to-citation path | Ask the vendor |
| Can you filter and export? | Stackable path filters, persistent filters, CSV export | Ask the vendor |
| Plan | Crawler logs start on Professional | Check the pricing page |
Source: Promptwatch llms.txt, ai-agent-instructions, and changelog, read November 3 and 4, 2026. We have not documented how other tools bucket bots, so we do not say.
Why the split matters
A search indexer visit means a crawler built an index that might later power answers. A citation crawl means a page was fetched at the moment someone asked. Those are different events, with different implications. A wave of indexer activity says the page is being read. A citation crawl says the page is being used in an answer.
One data point shows how fast the mix can shift. Meta-WebIndexer grew from about 2% to about 38% of tracked AI crawler requests, according to Promptwatch's data reports. If a tool counted all of those as one kind of visit, the line would jump. If a tool separated them, the jump would sit in a different row. Your reading of the chart depends on which.
Questions to ask any crawler-log tool
- Which bots are in "AI crawlers"? Ask for the list by name, not by category.
- Which counts as a citation or answer-time fetch? Ask whether search indexers are included.
- How are bots verified? A user agent string can be faked. IP verification against published operator ranges cannot be faked in the same way.
- Does a reclassification rewrite history? If the rules change, do old numbers change with them, or only new ones?
- Can I see the bot list per page? Page-level views tell an access problem from a content problem.
- What happens to unknown bots? Are they hidden, grouped as "other", or counted as AI?
- How do I export it? You may want the raw numbers in your own sheet.
Use the real bot names in any request. There is no bot called "ChatGPTBot"; Promptwatch tracks GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Agent, GoogleOther, Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher, and MistralAI-User.
What Promptwatch adds on top
Agent Analytics shows real-time, IP-verified AI crawler logs, a crawl-to-citation path with a citation rate per page, top pages, crawl discovery, and error tracking with failure explanations. A page with many crawls and a low citation rate points to a content problem. A page with few crawls points to access. That reading is the reason to keep the two buckets apart.
On November 4, sitemap page tabs also added an Agent Analytics tab per URL, with citation visits, crawler visits, and crawler logs for that page.
What it cannot tell you
A citation crawl does not mean you were named in an answer. It means a page was fetched. The citation itself shows up in citation analytics. Likewise, a high count of AI crawls does not mean better visibility. It means more reads.
Who should pick which
- A team that only needs a quick look at bot traffic: any log tool that names its bots and verifies IPs will do.
- An SEO lead explaining a chart change: a tool that documents its classification, so you can say why the totals moved.
- A content team deciding which pages to fix: per-page citation rates, which need the two buckets kept separate.
- A technical team debugging access: logs with failure explanations and CDN or server connections.
- An agency reporting to clients: a tool that lets you export and filter without a manual rebuild.
The recommendation
Treat the bucketing as a feature, not a footnote. Promptwatch is the one to evaluate first, because it documents the split, verifies bots by IP, and connects crawls to citations and to AI-referred traffic in the same product, while most trackers show only mentions. Start the 7-day free trial and connect your logs, or book a demo to walk through the classification on your own site.