Which software analyzes AI crawler logs from a CDN? Tools compared
Compare the software that analyzes AI crawler logs from your CDN, from Cloudflare's built-in dashboard to Promptwatch, Known Agents, Scrunch AI and enterprise log tools.
Your CDN already sees every request from GPTBot, ClaudeBot, PerplexityBot and the rest. The raw data is sitting in edge logs. The problem is turning millions of log lines into an answer to a plain question: which AI crawlers are visiting, which pages do they read, and does any of it lead to a citation?
Three kinds of software handle that, and they solve different problems. This guide sorts them out, ranks the options, and explains when a free dashboard is enough.
Three kinds of tool
CDN-native dashboards. Cloudflare's AI Crawl Control is the clearest example. It lives in the Cloudflare dashboard, needs no setup, and shows which AI crawlers request your pages. It only sees traffic that passes through that CDN.
Bot analytics products. These ingest logs or server-side events from your stack and report on crawlers and AI agents. Known Agents is the specialist here.
AI visibility platforms with crawler logs. These join crawler data to what AI engines actually say and cite. The value is in the join: a crawl by itself tells you a bot fetched a page, not whether anyone was told about it.
A fourth group, enterprise technical SEO suites such as Botify, treat log-file analysis as one part of a crawl and feed platform. They are powerful and sold through demos.
Ranking at a glance
| Rank | Tool | Crawler log capability | Price noted |
|---|---|---|---|
| 1 | Promptwatch | IP-verified real-time crawler logs, crawl-to-citation path, citation rate per page, error tracking | Crawler logs from Professional $245/mo; every agency plan from $199/mo |
| 2 | Scrunch AI | AI agent and bot traffic reporting | Core $250/mo; Enterprise custom |
| 3 | Profound | Agent Analytics alongside answer tracking | Enterprise custom, about $40k/yr per earlier estimate |
| 4 | Known Agents | Agent Analytics, spoof verification, referral tracking | Free; $29/mo; $299/mo |
| 5 | Cloudflare AI Crawl Control | Per-crawler requests, status codes, paths, robots.txt violations | Free on every Cloudflare plan |
| 6 | Botify | Log-file analysis inside an enterprise platform | Custom |
AthenaHQ is a strong tracker, but its catalog entry lists no AI crawler-log product, so it is not a crawler analytics answer.
1. Promptwatch
Promptwatch calls its crawler product Agent Analytics. It reads real-time, IP-verified logs, which means spoofed user agents are filtered out and what remains is traffic from the actual operators. Tracked bots include GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Agent, GoogleOther, Meta-WebIndexer, Meta-ExternalAgent, Meta-ExternalFetcher and MistralAI-User.
The CDN side is where it earns this ranking. Logs can come in through Cloudflare (Logpush or an auto-deployed Worker), AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP endpoint or the WordPress plugin. You are not locked to one CDN.
Then the analysis. You get a crawl-to-citation path with a citation rate per page, error tracking with explanations for failures, stackable path filters and CSV export. The 360 Insights view joins crawler logs, citation clicks and AI-referred visits per page, so a page that gets crawled and never cited stands out.
On plans: Essential has no crawler logs. They start on Professional ($245/mo, 25M logs), Business ($579/mo) allows 100M, and every agency plan includes them, starting at Kick-off ($199/mo, 10M logs). The 7-day trial is on monthly billing only.
2. Scrunch AI
Scrunch lists AI agent and bot traffic reporting on its Core plan ($250/mo) next to visibility tracking over four LLMs. It also sells an Agent Experience Platform that serves AI-optimized page versions to crawlers. The catalog notes reviewers dislike its reporting and that data refreshes weekly. A reasonable pick if serving separate pages to bots appeals to you.
3. Profound
Profound's Agent Analytics is part of an enterprise platform that has grown into marketing agents for content, demand and PR. The entry puts Enterprise around $40k a year, with a 7-day trial that tracks three engines. Fine for a large company that wants one vendor. Overkill if you only need crawler data.
4. Known Agents
Known Agents (formerly Dark Visitors) is the specialist: crawler and AI agent analytics, AI chat referral tracking, an auto-updating robots.txt and an agent identification API. It ingests data from WordPress, Cloudflare, Fastly, Akamai, AWS CloudFront, Google Cloud, Vercel and Node.js. Plans run Free (100K events a month), Startup $29/mo and Growth $299/mo. Two caveats from its own documentation: the Cloudflare Logpush route needs Cloudflare Enterprise, and citation counts are estimates inferred from fetches. It does not run prompts, so it cannot show what AI says about you.
5. Cloudflare AI Crawl Control
Free on every Cloudflare plan with zero configuration. It shows requests by crawler and operator, status codes, most requested paths and robots.txt violations, and you can allow or block a crawler from the same screen. The limits are real: it only covers traffic through Cloudflare, detection on the Free plan relies on user agent strings, and referral data needs a paid plan. No prompt tracking, no citations.
6. Botify
An enterprise technical SEO platform with log-file analysis, aimed at large sites and sold with a demo and custom quote. Strong where crawl budget and feeds are the job. It is not a replacement for tracking what AI engines answer.
Who should pick which
- You use Cloudflare and want a first look at no cost: switch on AI Crawl Control.
- You want crawlers plus citations, traffic and content in one place: Promptwatch, because a crawl without a citation or visit attached is hard to act on.
- You run several CDNs or a mixed stack: Promptwatch or Known Agents, both of which accept multiple sources.
- You want bot control and a robots.txt that updates itself: Known Agents.
- You are an enterprise SEO team already buying log analysis: ask Botify or Profound for a demo.
What crawler logs cannot tell you
Logs prove a bot requested a page. They do not prove an AI engine used it in an answer, and nothing guarantees a recommendation. That is why we put the platform with citation data first.