How do I use an llms.txt file, and does it help AI search visibility?

What an llms.txt file is

llms.txt is a proposed convention: a plain markdown file at the root of your site (yoursite.com/llms.txt) that gives language models a short, curated guide to what the site contains. Think of it as a hand-written table of contents for machines. It usually has the site or company name as a heading, a one- or two-sentence summary, and then sections of links with a line describing each page. Some sites also publish a longer companion, often named llms-full.txt, with fuller reference material.

The important word is "proposed". It is not a standard in the way robots.txt is a long-established convention that crawlers are built to honor. Publishing the file is cheap and harmless. Whether it changes anything about how assistants cite you is a different question, and we will be straight about it.

Does it help AI search visibility?

We cannot point you to a verified statement from the major search or AI vendors saying that their assistants read llms.txt and use it to choose citations, and we are not going to pretend otherwise. Claims that it "boosts visibility" tend to come from people selling generators or consulting. Treat them with the skepticism you would give any unmeasured claim.

That leaves a more useful framing:

• Cost is low. An hour of work, no technical risk if the file is accurate.
• Benefit is unproven. Any gain is hypothetical until you see a bot request the file.
• You can test it. If you have AI crawler logs, you can see whether any bot ever fetches /llms.txt. That is real evidence about your site, which beats any general claim.

What clearly does matter is whether AI crawlers can fetch your actual pages, and whether those pages answer the questions buyers ask. llms.txt cannot rescue a site that blocks bots or has thin content.

How to write one

1. Name and summary. One heading with your brand name, then a short summary stating what you do and for whom, in plain words.
2. Group your best pages. Sections such as Products, Pricing, Documentation, Guides, Company. Only link pages you want treated as authoritative.
3. Describe each link. A short phrase on what the page contains. Avoid marketing language.
4. Keep facts accurate and current. Pricing, plans and product names in this file should match your site exactly. A contradictory file is worse than no file.
5. Publish at the root and make sure it returns a normal 200 response, as plain text, not behind a login or a script.
6. Maintain it. Put it on the same review cycle as your pricing page.

You can write it by hand or use a generator. Promptwatch offers free llms.txt and robots.txt generators among its free tools, along with an Agentic Readiness Check that looks at how ready your site is for AI agents. Promptwatch also publishes its own llms.txt and a longer llms-full.txt, which shows the format in practice.

robots.txt matters more

robots.txt is the file crawlers are built to check. Different bots do different jobs: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Agent and others each have their own name. Whether you allow or block each one is a business decision. Allow the ones tied to search and citation if you want to be recommended. Blocking everything by default, or having a CDN or firewall block bots you did not intend to, is a common way to disappear from AI answers.

How to test whether llms.txt does anything for you

1. Publish the file.
2. Watch AI crawler logs for requests to /llms.txt, by bot.
3. Note the date, and track your citations for relevant prompts before and after.
4. Do not change anything else at the same time, so you can attribute any shift.

Promptwatch supports this with Agent Analytics: real-time, IP-verified crawler logs (spoofed user agents are filtered out), stackable path filters so you can isolate /llms.txt, error tracking with failure explanations, and a crawl-to-citation path with a citation rate per page. Citation Trends over time then shows whether anything moved. Crawler logs start on the Professional plan ($245 a month), and logs can come from Cloudflare, AWS CloudFront, Fastly, Vercel, Netlify, Akamai, Google Cloud CDN, a custom HTTP source or the WordPress plugin.

How other tools handle crawler visibility

AthenaHQ tracks AI answers and builds an action queue. Its catalog entry lists no crawler-log product, so you could not see whether a bot reads your llms.txt.

Scrunch AI includes AI agent and bot traffic reporting, and its Agent Experience Platform serves AI-optimized page versions to crawlers at the Enterprise tier. Core is $250 a month.

Profound pairs Answer Engine Insights with Agent Analytics. Enterprise contracts start around $40,000 a year per our existing entry.

Qwairy places crawler analytics on its Business plan, listed at €449 a month in our catalog.

What to do this week

• Publish an accurate llms.txt. It is a small job.
• Audit robots.txt, CDN and firewall rules for the bots you want.
• Use crawler logs to check whether bots reach your key pages without errors.
• Check, with a before-and-after, whether llms.txt requests ever appear.
• Spend the bigger effort on content that answers buyer prompts.

Who should pick which

• You want crawler logs, citation data and traffic attribution together: Promptwatch.
• You mainly want bot reporting and crawler-side serving: Scrunch AI.
• You are enterprise and want an agent program: Profound.
• You want to keep tracking but crawler logs are secondary: AthenaHQ.