llms.txt: What It Is, What the Evidence Says, Whether to Bother

What is llms.txt designed to do?

llms.txt (see also the glossary entry) is a proposed standard: a /llms.txt markdown file at a site's root that gives AI agents a curated, LLM-friendly guide to a site's content, rather than requiring them to parse full HTML pages to find what matters. Jeremy Howard first published the proposal on 3 September 2024, with a second version released 10 August 2026. The page frames it as solving a real, specific problem — LLMs have limited context windows and struggle to convert complex HTML into clean text, so a curated markdown summary is, in principle, easier for an agent to use efficiently.

Adoption of the format itself is real: the proposal's own page states that thousands of sites publish an llms.txt file, that some documentation platforms generate one automatically, and that OpenAI, Anthropic and Google all publish llms.txt files for their own developer docs. That's evidence of adoption as a convention. It is not, on its own, evidence that publishing one changes whether an AI engine cites or recommends a site — those are two different claims, and the second one is where the uncertainty lives.

Which engines fetch it?

Rarely, on the best independent evidence available. Ahrefs analyzed 137,210 domains in May 2026 using its own web analytics and bot-traffic data: 28% of domains (38,360) had published a valid llms.txt file, and 97% of those files received zero traffic that month. Of the requests that did reach published files, named AI bots accounted for only about 19.5% — GPTBot alone made up roughly 4.5% of fetches, with PerplexityBot and OAI-SearchBot fetching negligibly. The rest of the traffic to llms.txt files came overwhelmingly from non-AI tools: SEO audit crawlers, general web crawlers, and people or tools specifically studying llms.txt adoption itself, not consuming it for a real answer. Ahrefs' own framing of the finding: "Zero requests came from AI bots for llms.txt files that don't exist. They never go looking."

That's a large, dated, methodologically transparent sample, and it points the same direction as Google's explicit statement below: the major AI crawlers mostly aren't seeking this file out.

What does Google's guide say?

Unambiguously. Google's guide to optimizing for its generative AI features states that Google Search itself "doesn't use" llms.txt files, and that they "neither harm nor help your site's visibility or rankings." The same guide also debunks two related tactics sometimes bundled with llms.txt advice: there's "no requirement to break your content into tiny pieces for AI to better understand it," and no special machine-readable format is necessary for visibility in Google's AI features generally.

That statement is specific to Google Search. It doesn't speak for OpenAI, Anthropic or Perplexity, none of which have published an equivalent statement either confirming or denying that they use llms.txt for ranking or retrieval purposes. The Ahrefs crawler-traffic data above is the closest thing to an answer for those engines, and it points toward "rarely fetched, not clearly tied to citation outcomes" rather than toward active use — the same split between a crawler's training, search and user-triggered jobs that makes blanket "block all AI bots" robots.txt advice unreliable too; see blocking GPTBot isn't blocking ChatGPT.

What is the cost and risk of adding one?

Low cost, no identified downside, no demonstrated upside. Writing a basic llms.txt — a markdown summary of your site's structure and key pages — takes roughly an hour for most small-to-midsize sites, and nothing in the evidence above suggests it can hurt your visibility or rankings on any engine. The honest cost isn't risk; it's opportunity cost if a team treats it as a meaningful GEO lever and deprioritizes the things the evidence actually supports — crawlable pages, direct-answer structure, accurate facts — in favor of a file that, per the data above, mostly goes unread.

If you want to add one anyway — reasonable, given the near-zero downside — the llms.txt Generator drafts a starter file from your homepage navigation in a few seconds, for editing before you publish it.

What would change our verdict?

A study showing AI engines meaningfully using llms.txt content in a way that changes citation or recommendation outcomes — not just adoption counts, but a measured link to what actually gets said in an AI answer. Nothing we've found does that yet; the Ahrefs data measures whether the file gets fetched at all, which is a precondition for impact, not evidence of impact itself. If OpenAI, Anthropic or Perplexity publish documentation confirming they read and act on llms.txt files for ranking or retrieval, or if an independent study ties llms.txt presence to a measurable citation difference, that would move this verdict from "unproven" toward "worth prioritizing" — and we'd update this post and say so in the change log.

FAQ

Does llms.txt help my site rank better on Google? No, according to Google's own guidance: it explicitly states llms.txt files don't affect Google Search visibility or rankings either way.

Do any major AI engines actually read llms.txt files? Rarely, based on the best available crawler-traffic data. An Ahrefs study of 137,210 domains found 97% of published llms.txt files got zero traffic in a month, with named AI bots accounting for a minority of what traffic did arrive.

Should I still add an llms.txt file? It's a reasonable, low-cost thing to add if you have an hour — nothing in the evidence suggests it hurts you. Just don't treat it as a meaningful lever for AI visibility, and don't deprioritize better-evidenced work (crawlability, direct-answer structure, accurate facts) to make time for it.

Is llms.txt the same thing as robots.txt? No. robots.txt controls crawler access — what bots are allowed to fetch at all. llms.txt is a proposed content summary file, aimed at making a site's content easier for an AI agent to parse once it does have access.

What to do next

1. If you want one, generate a starter llms.txt with the llms.txt Generator and spend the real time reviewing and editing it, not treating the draft as finished.
2. Don't reprioritize crawlability, content structure or fact accuracy to make room for this — the evidence supports those far more strongly.
3. Revisit this verdict if a major AI vendor publishes documentation confirming active use of llms.txt for ranking or retrieval.