llms.txt: what it can do for you, what it cannot, and what to measure instead
An honest explainer on llms.txt: how the file is meant to work, why it is no substitute for crawler access and citations, and how to check whether it changes anything for your site.
Quick verdict
Publish an llms.txt if it takes you an hour, because a clear summary of your site is cheap and harmless. Do not expect it to replace crawler access, good page structure, or citations earned elsewhere. And do not trust anyone who claims a measured lift from it without showing logs. The honest way to find out whether it matters for you is to measure bot visits and citations before and after, which is a job for Promptwatch crawler logs and citation analytics.
What the file is
llms.txt is a plain text file at the root of a site, typically written in markdown. It lists the pages a site owner thinks a language model should read first, with short descriptions. Think of it as a curated table of contents for machines, sitting alongside robots.txt and the sitemap.
Promptwatch itself publishes two: an llms.txt and an expanded llms-full.txt that carries the full plan matrix, tracked models, and API catalog. They are a fair example of the format used well, because they carry facts a model would otherwise have to guess from marketing pages.
What it is not
It is not a permission system. Robots.txt tells bots what they may fetch. llms.txt does not grant or deny anything.
It is not a ranking signal that anyone has publicly confirmed. Whether a given AI product reads the file, and how heavily it weights it, is not something we can state for every engine. Treat claims otherwise as marketing.
It is not a fix for a blocked crawler. If a firewall returns 403 to OAI-SearchBot, a beautifully written llms.txt sits unread behind the same wall.
What it can plausibly help with
Three honest benefits, none of them magic.
Clarity for agents. Tools that fetch pages on behalf of a user, such as ChatGPT-User or Claude-User, may find a concise index faster than they would navigate a heavy site.
A forcing function for your own team. Writing the file makes you decide which twenty pages actually represent the business. Many sites discover that their best content is buried.
A place for stable facts. Pricing, product names, and company details in one reliable file reduce the chance that a model assembles them from stale third-party pages.
How to write a useful one
- Start with one paragraph describing the business in plain language.
- List core pages in groups, each with a one-line description of what a reader finds there.
- Include a pricing or facts page, kept current.
- Add a link to the fuller reference if you maintain one.
- Date it. A file that says when it was last updated is easier to trust.
Promptwatch has a free llms.txt generator and a robots.txt generator, plus an Agentic Readiness Check. They are a reasonable starting point if you want a draft without writing from scratch.
How to test whether it did anything
This is the step most posts skip.
Before publishing, record two baselines for the pages listed in the file: how often AI crawlers fetch them, and how often they are cited. After publishing, wait several weeks and compare. Changes in fetches could come from the file, or from a dozen other things, so change nothing else that month if you can help it.
| Measure | Where to get it | What a real effect looks like |
|---|---|---|
| Fetches of listed pages | Crawler logs, such as Agent Analytics | A sustained rise in named bots reading those URLs |
| Whether anything requests the file itself | Crawler logs filtered by path | Requests to /llms.txt from AI user agents |
| Citation rate for listed pages | Citation analytics, Page Tracker | More answers citing those pages |
| Visits | Visitor analytics | More AI-referred sessions on those pages |
If bots never request the file, you have your answer for that period. Agent Analytics supports stackable path filters, so filtering for the llms.txt path takes a moment, and CSV export is available if you want to chart it yourself.
Priorities that move the needle more
If you have a day to spend and the choice is between llms.txt and these, spend the day on these:
- Access. Make sure the search and fetch bots can reach your pages, in robots.txt and in the firewall.
- Page structure. Put the direct answer near the top of the page, with a clear heading that matches the question.
- Freshness. Update pages with outdated figures. Content Refresh in Promptwatch suggests diff-style updates for existing pages.
- Third-party presence. Check offsite mentions, Reddit, YouTube, and LinkedIn, where models often look for corroboration.
Plan notes
Crawler logs, the feature needed for the fetch measurements, start on Professional at $245 a month. Essential at $95 does not include them. Citation analytics and visitor analytics are available from Essential. Agency plans all include crawler logs from $199 a month.
Who should pick which
| You are... | Recommendation |
|---|---|
| A small site with spare time | Write the file, it costs little |
| A large site with firewall layers | Fix access first, file second |
| Under pressure to show a result | Run the before-and-after test with crawler logs |
| An agency asked "should we do this?" | Yes as hygiene, no as a promised lever, and show clients the log evidence |
| Evaluating tools | Prefer one that reads logs, such as Promptwatch, so claims can be checked |
More platforms are listed in the ranking. The file is a footnote. Measurement is the chapter, and the platform that records bot requests and citations side by side will tell you more in a month than any opinion piece will, this one included.