llms.txt: what it is, examples, and whether it matters

llms.txt explained: the format with a full example, llms-full.txt, who actually reads it (Google does not), when to publish one, and how to generate and validate it.

llms.txt is a proposed plain-text file, published at the root of a website, that gives large language models a short Markdown summary of the site and a curated list of links. It is cheap to make, harmless to publish, and, as of late 2026, almost nobody requests it. This reference covers the format with a complete example, the llms-full.txt convention, who actually reads these files according to the evidence, when publishing one is worth ten minutes, and how to generate and validate one without introducing mistakes.

The short version, for readers who need a decision before the detail:

Question	Answer	Basis
What is it?	A Markdown file at `/llms.txt` with an H1, a summary and link lists	The proposal at llmstxt.org
Does Google use it?	No. Google says Google Search ignores it; it neither helps nor harms rankings	Google’s AI optimisation guide, June 2026
Do ChatGPT, Claude or Perplexity use it?	None documents using it for retrieval or ranking	The operators’ crawler documentation
Who requests it?	Mostly SEO audit tools and general crawlers; AI retrieval bots made about 1 percent of requests	Ahrefs study of 137,210 domains, May 2026
Who publishes it?	Documentation platforms generate it automatically; Anthropic’s developer docs serve one	Mintlify docs; platform.claude.com/llms.txt
Should you publish one?	Yes if it costs minutes, because it is useful to agents pointed at it; no if you expect traffic from it	Our view

What is llms.txt?

llms.txt is a proposal by Jeremy Howard, first published in September 2024 at llmstxt.org, for a Markdown file at /llms.txt that tells language models and agents what a site is and which pages matter. It exists because HTML pages carry navigation, scripts and boilerplate that waste a model’s limited context, while a curated Markdown index does not.

The proposal defines a structure strict enough to parse programmatically and loose enough to write by hand. The file can sit at the site root or at any subpath, where it describes only the pages beneath that path; when several files apply, the proposal says an agent should use the most specific one. The llms.txt glossary entry gives the one-paragraph definition; the rest of this page is the specification in practice.

The format, section by section

The proposal lists five parts in a fixed order. Only the H1 is required.

Order	Element	Required	What goes in it
1	Byte-order mark	No	Optional; most generators omit it
2	H1	Yes	The site or project name. The only required element
3	Blockquote	No	A short summary with the key facts needed to understand the rest of the file
4	Free Markdown	No	Zero or more paragraphs, lists or other Markdown, but no headings, giving context or instructions on how to read the links
5	H2 file lists	No	Zero or more sections, each an H2 heading followed by a Markdown list where every item is a link, optionally followed by a colon and a note

Two conventions sit on top of that structure. An H2 named exactly Optional marks links an agent can skip when it needs a shorter context. And each link should point at content a model can read easily, which in practice means Markdown versions of pages where they exist (many platforms serve page.md beside page.html).

Here is a complete, valid example for a fictional analytics product. Every line maps to a row in the table above.

# Ledgerlight

> Ledgerlight is a self-hosted web analytics tool for small publishers.
> It stores no personal data, needs no cookie banner, and ships as a
> single binary with a PostgreSQL dependency.

The documentation below is written for engineers installing and
operating Ledgerlight. Pricing and company pages are listed under
Optional and are rarely needed to answer technical questions.

## Docs

- [Installation](https://ledgerlight.example/docs/install.md): System requirements, binary and Docker installs, first-run checklist
- [Configuration reference](https://ledgerlight.example/docs/config.md): Every environment variable with defaults and examples
- [Tracking script](https://ledgerlight.example/docs/script.md): The 1 KB snippet, custom events, and single-page-app routing
- [Privacy model](https://ledgerlight.example/docs/privacy.md): What is collected, what is hashed, and how retention works

## API

- [REST API](https://ledgerlight.example/docs/api.md): Authentication, endpoints, pagination and rate limits
- [Webhooks](https://ledgerlight.example/docs/webhooks.md): Event payloads and retry behaviour

## Optional

- [Pricing](https://ledgerlight.example/pricing): Plans for the hosted edition
- [Changelog](https://ledgerlight.example/changelog): Release notes by version
- [About](https://ledgerlight.example/about): Company and contact details

A few details in that example are deliberate. The blockquote states the three facts a model most often needs (what it is, the privacy position, how it deploys). The free-text paragraph tells an agent what to skip. Every link note is a sentence fragment describing what the page answers, not a marketing line. And the links point at Markdown files where the site has them, because a model given a choice between install.md and install.html gets more signal per token from the first.

Where the file lives

Serve it at https://your-domain/llms.txt with a text/plain or text/markdown content type and a 200 status. The proposal also allows subpath files such as /docs/llms.txt for a section with its own scope. Mintlify, which generates the file for every documentation site it hosts, additionally serves it at /.well-known/llms.txt and /.well-known/llms-full.txt (vendor documentation), so an agent that checks either location finds it. If you only publish one location, make it the root.

What is llms-full.txt?

llms-full.txt is a convention, not part of the llmstxt.org proposal, for a single file that concatenates the full text of every documentation page in Markdown. Where llms.txt is an index an agent navigates, llms-full.txt is a dump an agent loads whole, which suits assistants that accept a pasted file or a URL as context.

The convention comes from documentation platforms. Mintlify describes its generated llms-full.txt as combining the entire docs site into one file, with each page’s title, source URL, description and full Markdown content, following the same visibility rules as its per-page Markdown views (Mintlify’s llms.txt documentation). The files can be large; a mature API’s llms-full.txt runs to hundreds of thousands of tokens, which is more than most models accept in one request. That is why the index exists: the index fits, the dump often does not.

Publish llms-full.txt only if you have a generator that keeps it current. A hand-maintained copy of your documentation drifts within a month, and an agent that reads a stale dump will answer confidently with outdated instructions. The llms-full.txt entry has the one-paragraph version.

Who actually reads llms.txt?

The honest answer is: tools that audit sites, general crawlers, some coding agents and documentation assistants when they are pointed at the file, and almost no AI search engine on its own initiative.

Reader	What is documented	Status
Google Search, AI Overviews, AI Mode	Google says: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search” because “Google Search itself doesn’t use them”, and that maintaining an llms.txt “will neither harm nor help” visibility or rankings (Google’s AI optimisation guide). Google added that clarification on June 15, 2026 (Search Central updates)	Not used
ChatGPT search (OAI-SearchBot), Claude (Claude-SearchBot), Perplexity (PerplexityBot)	Each operator documents its crawlers, their purposes and how to block them. None documents reading llms.txt for retrieval, indexing or ranking	No documented use
Bing and Copilot	Bing documents its crawler and the `noarchive` and `nocache` controls for generative answers; nothing about llms.txt	No documented use
Coding agents and documentation assistants	Platforms such as Mintlify and GitBook generate the files for hosted docs; llmstxt.org lists plugins for VitePress, Docusaurus, Drupal and others, and an MCP server that reads the file. Anthropic serves one for its own developer documentation (platform.claude.com/llms.txt)	Used when pointed at it
SEO audit tools and general crawlers	The main consumers of the file in the Ahrefs data below	Fetch it; do nothing with it

Research shows how small the real readership is. Ahrefs analysed 137,210 domains that received traffic in May 2026, found that about 28 percent served a valid llms.txt, and that 97 percent of those files received no requests at all that month, from bots or humans. Of the requests that did arrive, 96 percent came from bots: SEO audit tools made 21.7 percent, general web crawlers 13.1 percent, and AI retrieval bots about 1.1 percent of the total. The authors concluded the file is “largely decoration” for appearing in ChatGPT, Perplexity or AI Overviews, that its likeliest readers are coding agents, and that AI systems do not go looking for a missing file, so omitting one costs nothing (Ahrefs’ llms.txt study). The study’s caveats are fair: its sample skews towards technical sites, it did not check whether files followed the spec, and a fetch does not prove a bot used what it fetched.

Our audit scores llms.txt as informational for exactly this reason: one point, labelled optional, with the Google and Ahrefs findings in the finding text.

Does llms.txt help SEO?

No, on the evidence available. Google says its search does not use the file and that publishing one neither helps nor harms rankings. No AI search operator documents using it. The only measurable effect is on agents that are pointed at the file by a person or a tool, which is a documentation benefit, not a search benefit.

That does not make it worthless; it makes it a different kind of asset. The case for publishing one rests on three things that are true today:

It costs minutes. A generator writes the file from your sitemap or navigation, and an index of thirty links needs updating a few times a year.
Agents are growing as a traffic source. Ahrefs’ data found agentic infrastructure at 10.5 percent of llms.txt requests, more than any AI search bot, and a developer who asks a coding assistant “how do I configure X” is a customer you want answered correctly.
It is the cheapest place to state what your site is. The blockquote is a one-paragraph positioning statement that a model will quote back if it reads the file at all.

The case against is equally plain: if anybody tells you llms.txt moved their rankings or got them cited in AI Overviews, the claim contradicts Google’s documentation and has no published evidence behind it. Treat the file as you would a humans.txt: a courtesy, not a channel.

When is it worth publishing?

Use this table to decide in under a minute.

Your situation	Publish?	Why
Developer documentation, an API, a CLI or an open-source project	Yes, both files, generated	Coding agents are the readers that exist, and docs platforms make it free
A SaaS product with help content	Yes, llms.txt only	An index of the help centre and the product’s positioning statement helps assistants answer support questions
A publisher or blog	Optional	Few readers; a thirty-link index of your best guides costs nothing and may help an agent pointed at your site
A local business or brochure site	Optional	A five-line file with your services and contact page is harmless; expect nothing from it
An ecommerce catalogue	Probably not	Product data belongs in feeds and structured data; a Markdown list of 40,000 products is not an index
Anyone hoping to rank or be cited because of it	No	Google ignores it; nobody else documents using it

If you publish, also decide whether to let search engines index the file itself. Google’s staff have suggested there is no reason for it to appear in results; an X-Robots-Tag: noindex header on /llms.txt keeps it out of the index without stopping anything from reading it.

How to generate and validate an llms.txt file

A correct file takes four steps. The llms.txt generator does the first three from a URL and runs the validation checks in the fourth.

Write the H1 and the blockquote by hand. The site name, then two or three sentences a stranger would need: what the site is, who it is for, and the one fact that is most often misunderstood about it.
Choose the links. Twenty to fifty pages that answer questions, grouped under two to five H2 headings. Prefer reference pages, definitions and how-to guides over landing pages. If a page has a Markdown version, link that.
Write a note for each link. One fragment stating what the page answers. “Pricing” is a label; “Plans, limits and what happens when you exceed them” is a note.
Validate before and after publishing. The checklist below is what a validator should test, and it is what our generator reports.

Validation checklist:

The last item deserves a sentence. Because the file is designed to be read by models, it is also an obvious place for prompt injection. Ahrefs raised the same concern. Keep it factual, keep it reviewable in version control, and never let a third party write to it.

To confirm that the crawlers you care about can fetch the file and your pages, run the AI visibility checker, which shows the crawler table and whether llms.txt was found, and the robots.txt tester for the specific user agents. Note that robots.txt, not llms.txt, is the file that actually controls crawler access; the AI crawlers guide explains every agent and the policy choices.

Common mistakes

Mistake	Why it matters	Fix
Publishing an HTML page at `/llms.txt`	Parsers expect Markdown; our audit flags a file that does not start with an H1 as invalid	Serve plain text or Markdown with a text content type
Dumping the sitemap into the file	A thousand bare URLs with no notes is not an index; it is noise an agent must skip	Curate 20 to 50 links with notes; use llms-full.txt for bulk content
Writing marketing copy in the blockquote	A model quotes the summary; superlatives make it less useful and less likely to be trusted	State facts: what, for whom, how deployed, what it costs
Linking HTML pages when Markdown versions exist	More tokens, less content	Link the `.md` version where available
Letting it go stale	A stale index points agents at removed pages and old prices	Generate from navigation or sitemap; review quarterly
Blocking the crawlers you hope will read it	robots.txt rules apply to the file too	Test `/llms.txt` with the robots.txt tester
Expecting rankings or citations from it	Google ignores it; no operator documents using it	Spend the time on indexable, snippet-eligible pages instead
Including instructions to models	Prompt-injection risk and a trust problem if discovered	Keep it descriptive

What to do next

Generate a file with the llms.txt generator, check it against the validation list, and publish it with a noindex header. Then spend the rest of the hour on the things that do determine whether AI engines cite you: crawler access in robots.txt and your firewall, content that is present in the raw HTML, and pages that are indexed and snippet-eligible. The free audit at the top of this site checks all of those for a URL in about a minute and reports llms.txt as the optional item it is. For the retrieval side, how generative engine optimisation works and the AI visibility guide cover what the operators actually document.

Frequently asked questions

Does llms.txt help SEO?: Not for Google. Google's documentation says Google Search does not use llms.txt or similar AI text files, and that creating one will neither help nor harm your visibility or rankings. No other major search or AI operator has documented using it for retrieval or ranking either. It can help coding agents and documentation assistants that are pointed at it directly.
Does Google use llms.txt?: No. Google's AI optimisation guide says you do not need to create machine-readable files, AI text files, markup or Markdown to appear in Google Search, because Google Search itself does not use them, and that maintaining an llms.txt for other services is fine.
Where should llms.txt go?: At the root of the site, as /llms.txt, served as plain text or Markdown. The proposal also allows a file at a subpath, which then covers only the pages under that path. Some platforms additionally serve it at /.well-known/llms.txt.
What is the difference between llms.txt and llms-full.txt?: llms.txt is a short Markdown index: a title, a summary and curated links with one-line descriptions. llms-full.txt is a convention used by documentation platforms to concatenate the full text of every page into one large file. The first is an index for an agent to navigate; the second is a dump for an agent to load whole.
How do I validate an llms.txt file?: Check that it starts with a single H1, that an optional blockquote summary follows, that every file list is an H2 section containing Markdown links with optional notes, that an Optional section is named exactly that, that the file is served with a text content type and a 200 status, and that every linked URL returns 200.
Should llms.txt be indexed by Google?: There is no benefit to it appearing in search results, and a plain text file can occasionally show up for odd queries. Serving it with an X-Robots-Tag noindex header keeps it out of the index while leaving it readable by anything that requests it directly.