Is llms.txt required for AI search optimization? What to do instead of guessing
Is llms.txt required for AI search optimization? Why you should test it with crawler logs rather than assume, and which tools can show whether bots read it.
Short answer: no evidence we can point to says llms.txt is required, and you should not treat it as a ranking switch. It is a proposed convention, a plain markdown file at the root of a site that gives language models a curated map of your content. Some sites publish one. Whether any given engine reads it, and whether reading it changes what gets cited, is something you can measure on your own site instead of arguing about.
A caveat on sourcing: we have not quoted any engine's policy here. Google Search Central, OpenAI and Anthropic publish their own documentation on how their crawlers and products work. Read those for the current official position, because it can change.
What llms.txt is, and is not
It is a file at /llms.txt with a short description of your site and links to the pages you most want a model to use, often in markdown. A longer variant lists fuller content. It is a suggestion to machines, not an access control. It does not block anything, so it is not a replacement for robots.txt. And it is not a sitemap, though it can resemble one.
What it is not: a required file, a documented ranking factor, or a substitute for pages that are worth citing. If your pages do not answer questions well, a tidy index of them will not help.
Why people publish one anyway
- It is cheap. A competent person can write one in an hour.
- It forces you to decide which pages matter most.
- Some AI tools and developer workflows can use it as context, so documentation sites in particular find it useful.
- It may help in the future, and costs little to keep.
None of that makes it required. The honest framing is "low cost, uncertain benefit, measurable on your own site".
How to test it on your own site
- Publish the file.
- Watch your crawler logs for requests to
/llms.txt, and note which bots make them. - Compare citation behavior for the pages you listed before and after.
- Keep what works, drop it if nothing changes.
Step two is the one most people skip. If no real AI crawler ever requests the file, you have your answer for your site. If some do, you can see which.
Tools that help you test
1. Promptwatch
Promptwatch fits this test well. Agent Analytics provides real-time, IP-verified AI crawler logs (so spoofed user agents are filtered out) with stackable path filters, so you can filter to /llms.txt and see which of GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and others request it. It also has the crawl-to-citation path with a citation rate per page, so you can check whether the pages you listed are cited more over time. A free llms.txt generator and robots.txt generator are available among its free tools. Crawler logs start on the Professional plan ($245 a month); Essential ($95) does not include them. Explore is free (10 prompts, ChatGPT only).
2. Cloudflare AI Crawl Control
Cloudflare AI Crawl Control is included on every Cloudflare plan and shows which AI crawlers request your pages, grouped by operator, with most-requested paths, status codes and robots.txt violations. On Free, crawlers are identified by user agent only. If you are behind Cloudflare, it is a no-cost way to look for /llms.txt requests.
3. Ahrefs Brand Radar
Ahrefs Brand Radar reports on AI crawler activity as part of its AI visibility product. Custom Prompts is included on paid Ahrefs plans.
4. Scrunch AI
Scrunch AI reports on AI agent and bot traffic, and its Agent Experience Platform on Enterprise serves AI-optimized versions of pages to LLM crawlers, a different approach to the same goal. Core is $250 a month.
5. Goodie AI
Goodie AI lists fixing technical signals such as llms.txt and sitemaps among the jobs of its agents. Plans start at $295 a month.
6. AthenaHQ and Profound
AthenaHQ (Starter $295 a month) and Profound (custom Enterprise, estimated near $40k a year) focus on visibility and actions rather than crawler-path testing.
Comparison table
| Tool | Can show requests to /llms.txt | Price |
|---|---|---|
| Promptwatch | Yes, via crawler logs and path filters (Professional and up) | From $245/mo for logs |
| Cloudflare AI Crawl Control | Most-requested paths, if you use Cloudflare | Free on every plan |
| Ahrefs Brand Radar | AI crawler activity reporting | Free Custom Prompts with paid Ahrefs |
| Scrunch AI | Agent and bot traffic reporting | From $250/mo |
| Your own server logs | Yes | Free |
Who should do what
- Documentation or developer-tool sites: publish one. The cost is low and there are plausible consumers.
- Everyone else: publish a short, accurate one if you like, but spend the real effort on crawl access and content.
- Anyone who wants an answer: measure requests and citations rather than rely on opinion.
What matters more
Check robots.txt does not block the search-oriented AI bots you want. Make key pages render in plain HTML. Answer questions directly and specifically. Earn mentions on sources engines cite. Then measure citations and AI-referred traffic.