How to read a crawl-to-citation path and decide what to fix
A diagnostic guide for the page-level funnel from AI crawler fetch to citation to visit: what each stage means, the four common failure patterns, and the fix for each.
Quick verdict
Visibility reports tell you where you stand. A crawl-to-citation path tells you why. It follows a page through four stages: discovered, fetched, cited, and visited. A page that stalls at one stage has one kind of problem, and each stage has its own fix. Among the tools in our directory, Promptwatch is the one that publishes this path with a citation rate per page, because it holds crawler logs, citation data, and visitor analytics in the same workspace.
The four stages
Think of every page as a small funnel.
- Fetched. An AI crawler or agent requested the page and got a response.
- Readable. The response was a success, not an error, and the content was in the HTML the bot received.
- Cited. An AI answer used the page as a source.
- Visited. A person clicked the citation and landed on the page.
Most reporting stops at stage three, and most tools stop at stage one or three. The value of the full path is that the drop between two stages tells you where to look.
Pattern one: no fetches
If a page you care about shows no crawler visits, nothing downstream can happen. Causes are usually mundane:
- robots.txt blocks the relevant bots.
- A firewall or CDN bot rule rejects them.
- The page is not linked from anywhere a crawler would find it and is missing from the sitemap.
Fix: open access first, then add internal links and a sitemap entry. Check logs a week later to confirm fetches appear.
Pattern two: fetches with errors
The bot arrived and left empty-handed. A 403 points to a firewall. A 429 points to rate limiting. A 5xx points to server trouble under crawler load, and a redirect chain may point to a migration that was never cleaned up.
Promptwatch's Agent Analytics includes error tracking with failure explanations, which saves the step of decoding status codes yourself.
Fix: allowlist verified bot ranges, relax rate limits for known crawlers, and flatten redirects.
Pattern three: clean fetches, no citations
This one stings because the technical work is done. The bot reads the page, and the answer engine uses someone else.
Usual reasons: the page does not answer the question near the top, the content is thin compared with the pages that are cited, key information is in images or scripts that the bot does not render, or an authoritative third party already covers the topic better.
Fix: compare your page against whatever is being cited for the same prompt. Promptwatch's Page Tracker can follow a competitor's URL alongside yours. Restructure around the question, add specifics that only you can supply, and update stale numbers. Content Refresh produces diff-style update suggestions for existing pages, which suits this case.
Pattern four: citations, no visits
The page is cited and nothing arrives. Perhaps the citation sits low in the source list, the answer fully satisfies the question without a click, or the landing page does not match the intent.
This is the only pattern you cannot diagnose from crawl and citation data alone. You need visitor analytics, which records AI-referred sessions and conversions from ChatGPT, Perplexity, Gemini, Claude, Copilot, and more, through a script, a GTM template, or the WordPress plugin.
Fix: improve the page's pull for the click, such as a distinctive angle the summary cannot fully give away, and make sure the landing experience delivers what the citation promised.
Reading the numbers together
| Fetches | Errors | Citation rate | Visits | Diagnosis |
|---|---|---|---|---|
| Zero | n/a | n/a | n/a | Access or discovery |
| Some | High | Low | Low | Technical failure |
| Many | Low | Near zero | Near zero | Content mismatch |
| Many | Low | Healthy | Low | Click appeal or intent mismatch |
| Many | Low | Healthy | Healthy | Working page, protect it |
Promptwatch's 360 Insights joins crawler logs, citation clicks, and AI-referred visits per page, and its Topic Insights show where pages underdeliver for their topic. That is the table above, produced for you.
Citation Trends adds time
A snapshot can mislead, so add time. Citation Trends classifies each citation by content type and source type, and reports first cited, ramp-up days, peak rate, and decay half-life per page. A page that peaked and decayed fast is a refresh candidate. A page that ramped slowly and held is evergreen, so leave it alone.
What it costs to get this view
Crawler logs begin on Professional at $245 a month with 25M logs. Business ($579) raises that to 100M and adds the Content Agent. On the agency side, every plan includes crawler logs, from $199 a month. Essential at $95 has visitor analytics and citation data but no crawler logs, so the path would be incomplete there.
If you would like a feel for it before paying, the free Explore tier and the no-login AI Visibility Report show the visibility side, and your own CDN logs can stand in for the crawler side on a handful of pages.
Who should pick which
| You are... | Recommendation |
|---|---|
| Diagnosing one page that vanished | Your own logs plus a manual citation check |
| Running a content program of dozens of pages | Promptwatch Professional, for the path per page |
| Needing proof for engineering or IT | Agent Analytics error tracking with CSV export |
| An agency with many client sites | Agency plans, crawler logs on all |
| Only tracking mentions | A monitoring tracker is fine, but you will not get causes |
The full ranking shows alternatives. The short version of our advice: if a tool cannot tell you which stage a page stalls at, it is a reporting layer, and you will end up guessing at fixes.