Depends what you expect it to do, and the honest answer has changed as real data arrived.
What it is
llms.txt is a proposed convention: a markdown file at your domain root listing the pages you consider most important, each with a short description, so a language model retrieving from your site can find the useful parts without crawling everything. A table of contents written for machines.
It is not a permissions file. robots.txt controls access; llms.txt only suggests priority. Adding one grants nothing and blocks nothing.
What the measurements show
Two large studies have now looked at this, and they agree on the thing that matters.
Ahrefs examined 137,210 domains that received traffic in May 2026. Among those publishing an llms.txt file, 97% of the files received no traffic at all that month. Only about 1,100 domains saw any requests whatsoever.
SE Ranking crawled roughly 300,000 domains and looked specifically for an effect on citations. They found no relationship between having an llms.txt file and how often a domain was cited in major LLM answers. Adoption was also strikingly flat across traffic tiers — 9.88% for low-traffic sites, 10.54% mid, 8.27% high — meaning the biggest sites are, if anything, slightly less likely to bother.
The adoption rates disagree — Ahrefs found 28%, SE Ranking 10.13%. That is a population difference rather than a contradiction: Ahrefs measured domains using its own web analytics product, a self-selected group of SEO-aware sites, while SE Ranking crawled more broadly. The realistic figure for "websites in general" is nearer the lower number.
Why those studies are weaker evidence than they look
The obvious reading of "97% never fetched" and "no citation effect" is that the format does nothing. That reading goes further than the data supports, and it is worth being precise about why.
Both studies measure the average published file, not a good one. The overwhelming majority of llms.txt files in the wild were generated automatically — a plugin, a build step, a checkbox — listing every page on the site with its meta description pasted alongside. Many sit on sites that retrieval systems were not fetching anyway. A study asking "does adding this file to a randomly chosen website change its citation rate" is asking a fair question, and the honest answer is no. That is not the same question as "does a carefully built file do anything for a site whose content is worth retrieving."
Meanwhile, look at who bothers. Stripe, Cursor, Cloudflare, Vercel and Anthropic all publish and maintain one. These are among the most technically careful organisations on the web, they do not ship files for decoration, and they have feedback nobody else has — they can see whether assistants answer questions about their own products correctly. Sustained adoption by that group is revealed preference, and on a question this new it is arguably stronger evidence than a correlation across 300,000 mostly unsophisticated domains.
One precision note, since a lot of guides get it wrong in the other direction: several state that Anthropic has confirmed Claude respects llms.txt in retrieval. We could not find support for that, and a June 2025 feature request on the Claude Code repository asking for llms.txt support suggests it was not implemented then. Anthropic publishing one is a documented fact; Claude consuming one is not. Worth keeping straight — though for the argument above it is the publishing that carries the signal.
So the two findings sit together without contradiction: the format is real and the people closest to it act like it works, and dropping a generated file onto an ordinary business site changes nothing measurable. Both are true, and the log data shows exactly why.
The number that reconciles them
Buried in the Ahrefs data is the finding that makes sense of the rest. Among the small share of llms.txt files that did receive requests, here is who was asking:
| Requester | Share of requests |
|---|---|
| SEO audit tools | 21.7% |
| Unidentified | 14.9% |
| General crawlers | 13.1% |
| Tech profiling tools | 11.6% |
| AI agents | 10.5% |
| AI retrieval bots (OAI-SearchBot, PerplexityBot) | 1.1% |
Read the last two rows together. AI agents fetch llms.txt roughly ten times as often as AI search retrieval bots do.
That is the whole story. An AI agent is a coding assistant, a research tool, something working through your documentation on a user's behalf right now. A retrieval bot is what builds the index behind an AI search answer. The format is being used by the first group and largely ignored by the second.
Which explains everything else on this page: why the companies with serious implementations are Stripe, Cursor, Cloudflare, Vercel and Anthropic — all documentation-heavy, all with users working alongside AI assistants — and why a study looking for a citation effect found nothing. It was looking in the wrong place. The file is doing a job; it is just not the job most businesses added it for.
llms.txt is being used by agents, not by search.
So what does that mean for you
If your site is documentation — an API reference, a developer tool, a technical library, a product with a large help centre — build one properly. Your users are pointing assistants at your docs, those assistants do fetch it, and a good file measurably reduces the wrong answers they give about your product.
If you want to be cited in AI search answers, the file is worth having and it is not what decides the outcome. It sits downstream of three things, and until those are true it has nothing to work with:
- Can a crawler reach you? Check that
OAI-SearchBot,ChatGPT-User,ClaudeBotandPerplexityBotare permitted by name inrobots.txt, then separately check whether bot protection on your host or CDN is turning them away regardless of what that file says. The second catches far more sites than the first, and it fails silently. - Is your content in the HTML? Several of these crawlers do not run JavaScript.
- Is there anything specific to quote? An assistant repeats verifiable facts and skips adjectives.
Get those right and a well-built llms.txt is a sensible finishing touch on work that will already be paying off. Get them wrong and it is a map to a building nobody can enter — which, given 97% of these files are never fetched, is roughly what has happened across the web.
The pattern the careful adopters share is worth copying regardless of your industry: they treat the file as a curated index of pages that genuinely answer something, and they keep it current. That is what separates the 3% that get fetched from the 97% that do not.
A quick way to tell if yours is any good
Ten minutes, and it works whether you have a file already or are deciding whether to write one.
Say you run a deck-building company covering three towns east of Seattle, and your site has fourteen pages. Two candidate entries for your llms.txt:
/services— Our services page
versus
/composite-decking— Composite and cedar deck construction in Bellevue, Kirkland and Redmond. Covers typical cost ranges, permit requirements, and build timelines.
The first is a filename with a label attached. The second tells a retrieval system what question the page answers and who it answers it for.
Now the harder half of the test, which is where this stops being about llms.txt at all: open that page and check the description is true. Most fail here. The line promises cost ranges and permit requirements — because that is what customers actually ask — and the page turns out to contain three paragraphs about craftsmanship and a contact form.
That mismatch is the useful finding. Writing a good llms.txt forces you to state, in one line per page, what each page actually delivers. Most businesses discover they cannot, because the pages do not deliver anything nameable. The file is easy. Knowing what belongs in it is the work — and doing that work fixes the pages, which is what was going to move the needle in the first place.
Two conditions if you add one
Keep it accurate. A stale llms.txt is worse than none, because whatever does read it will trust it. It drifts by default — nothing breaks visibly when a page is renamed, so nobody notices.
Do not mistake it for the work. The real risk is not the file; it is the feeling of having addressed AI visibility by adding one. That feeling is comfortable, cheap, and exactly where a lot of businesses stop.
Our Upgrade Blueprint is $299 and covers the whole path in order: whether crawlers are actually getting through, whether your pages can be read and quoted, which buyer questions your site currently fails to answer, and what a machine-readable layer over the top of that should say. Fully credited if we do the work.
More on the first step — including how to check the bot-protection problem yourself — in why an assistant may not know your business exists.
Sources
- Ahrefs — We Analyzed 137K Sites: 97% of llms.txt Files Never Get Read. Source of the 137,210-domain figure, the 28% adoption rate within that population, the 97% no-traffic finding, and the full requester breakdown (SEO tools 21.7%, unidentified 14.9%, general crawlers 13.1%, tech profiling 11.6%, AI agents 10.5%, AI retrieval bots 1.1%).
- SE Ranking — LLMs.txt: Why Brands Rely On It and Why It Doesn't Work. Source of the ~300,000-domain crawl, the 10.13% adoption rate, the per-tier adoption figures, and the absence of any citation effect.
- Search Engine Journal — LLMs.txt Shows No Clear Effect On AI Citations, Based On 300k Domains. Independent coverage of the SE Ranking study.
- Otterly.AI — The llms.txt Experiment. A separate 90-day server-log study: 84 of more than 62,100 AI bot visits targeted llms.txt, around 0.1%. Consistent with the two larger datasets.
- Google's position was stated by Gary Illyes in July 2025 and is reported consistently across search-industry coverage; we have not seen it published as formal Google documentation.
Figures checked 2026-07-31. This area moves quickly, and the platform positions above are the part worth re-checking before acting on any of it.