Why isn't my website showing up in AI search results? Eight causes and how to fix them
If your website isn't showing up in AI search results, the cause is usually blocked crawlers, missing indexing or weak third-party presence. Here is how to diagnose it.
You search for your own category in ChatGPT, Perplexity or Google's AI Overviews, and your site is nowhere. Competitors are named. You are not cited. It is frustrating, and it usually has an ordinary explanation. The causes fall into three groups: the engine cannot reach you, the engine can reach you but has no reason to pick you, or you are showing up and not noticing. Work through them in that order.
Group one: the engine cannot reach you
1. Your robots.txt blocks the crawler
This is the single most common cause. Search robots.txt for the bots that feed AI answers: OAI-SearchBot and GPTBot (OpenAI), ClaudeBot and Claude-SearchBot (Anthropic), PerplexityBot, Googlebot and Bingbot. OpenAI's documentation says sites that disallow OAI-SearchBot may not appear in ChatGPT search answers, so a blanket "block AI bots" rule added for training reasons can remove you from search results too.
2. A firewall or CDN is doing the blocking
Bot-protection settings often block unfamiliar user agents regardless of robots.txt. You will not see the problem in the file. You see it in server logs as 403 or 429 responses to AI bots.
3. Your pages are not indexed
Engines that search the live web rely on search indexes. Check Google Search Console and Bing Webmaster Tools. Copilot draws on Bing, so a site missing from Bing can be missing from Copilot answers. Look for noindex tags, canonical errors, orphaned pages and very slow responses.
4. Your content only exists after JavaScript runs
If the main text is injected client-side, some crawlers will see an empty shell. View the page source, not the rendered page, and check that the content is there.
Group two: the engine can reach you but picks someone else
5. Your page does not answer the question
Engines quote pages that answer directly. A page that opens with three paragraphs of context and buries the answer is harder to lift than one that states it in the first lines under a clear heading.
6. You are absent from the sources engines trust
AI answers often lean on third-party pages: forums, review sites, comparison articles, videos. If competitors are discussed there and you are not, engines have more material about them. Brand mentions on third-party pages count even without a link.
7. Your brand is ambiguous or new
If your name is a common word, or your brand is recent, models have little to anchor on. Consistent naming, clear "what we do" statements and a few credible external mentions help.
Group three: you may already be showing up
8. You are checking the wrong way
A single manual search proves very little. Answers vary between runs, by country, and by how the question is phrased. You may appear for some prompts and not others. Without a fixed prompt set tracked over time, you are guessing.
About llms.txt
Many sites add an llms.txt file hoping it will fix visibility. We know of no engine that commits to requiring it, so treat it as optional and never as the fix for causes one through four.
How to diagnose it quickly
| Tool | What it shows you | Entry price |
|---|---|---|
| Promptwatch | Crawler logs with errors, site audit, prompt tracking, citations, AI-referred traffic | Free Explore; paid from $95/mo |
| AthenaHQ | Prompt tracking and GEO score, Action Center | Free Essential; Starter $295/mo |
| Scrunch AI | Prompt tracking, site audits, bot traffic reporting | Core $250/mo |
| Profound | Enterprise answer analytics | Free trial; Enterprise custom |
| HubSpot AEO | Visibility and citation analysis on three engines | $50/mo |
Why Promptwatch suits this problem
Most trackers can tell you that you are absent. The harder question is why, and that is where crawler logs come in. Promptwatch's Agent Analytics shows IP-verified AI crawler activity for bots such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot, with errors and failure explanations per request and a citation rate per page. If OAI-SearchBot gets a 403 on your pricing page, you will see it.
Alongside that, a weekly Site Health audit runs 21 checks and crawlability tests. Citation analytics show which pages and third-party domains engines use for your prompts, and the Action Board turns findings into a prioritized list. Crawler logs start on Professional ($245 a month); the site audit and prompt tracking begin on Essential ($95).
For a quick first look at no cost, it also offers an AI Visibility Report (no login) and an Agentic Readiness Check.
Where the others fit
AthenaHQ and Scrunch AI will tell you where you are absent and help organize fixes. Scrunch AI includes five site audits a month on Core. HubSpot AEO is a low-cost way to see whether you are named on three engines. We found no published crawler-log product in those listings, which makes causes one and two harder to confirm.
A 30-minute diagnosis
- Open robots.txt and look for the bots listed above.
- Check server logs or your CDN dashboard for 403, 429 and 5xx responses to AI user agents.
- Verify the site in Google Search Console and Bing Webmaster Tools.
- View source on your key pages to confirm content is present without JavaScript.
- Write 20 prompts, run them across engines twice, and note where you appear.
- List which third-party sources engines cite instead of you.
What no one can promise
Fixing access removes the biggest obstacle. It does not guarantee that an engine will name you, because answers change between runs and models update. Fix the causes you can control and track the outcome.