llms.txt: the complete guide to the AI content standard

llms.txt is a plain-text file, written in Markdown and placed at the root of a website (yourdomain.com/llms.txt), that gives AI systems a short, curated guide to the site's most important content. A spec-compliant file contains the site's name as a heading, a one-paragraph summary in a blockquote, and grouped lists of links — each with a one-line description — pointing to the pages a language model should read to understand and represent the site accurately.

The problem it tries to solve is real. HTML is a hostile format for language models: navigation menus, cookie banners, scripts, ads, and boilerplate consume the model's limited context window before it ever reaches your actual content. Context windows are finite, and an AI system answering a question at inference time has seconds — not hours — to figure out which of your thousand pages matter. llms.txt proposes a fix at the source: the site owner publishes a pre-curated index, in a format models parse natively, pointing to clean content. The proposal pairs with a companion convention of offering Markdown versions of important pages, so a model can skip the HTML entirely.

Now the part you should hold onto for the rest of this guide: llms.txt is a proposal, not a ratified standard. There is a published spec anyone can implement, a genuine tooling ecosystem forming around it, and a growing number of sites shipping one. But no standards body has blessed it, and — more importantly — no major AI engine has confirmed that it uses llms.txt for retrieval or training. Both of those facts can be true at once: the format is real and well designed, and its consumption is unproven. Any guide that tells you only one half is selling you something.

llms.txt was proposed in September 2024 by Jeremy Howard, co-founder of Answer.AI and the deep-learning education project fast.ai. The spec lives at llmstxt.org, and the original argument is worth reading in full because it is narrower — and more honest — than most of the coverage it spawned.

Howard's case was not "this will make AI engines rank you higher." It was an engineering observation: language models are increasingly asked to use websites at inference time — answering questions about a product, helping a developer integrate an API, navigating documentation — and websites are built for browsers, not models. Converting a modern HTML page to something a model can use cleanly is lossy and expensive, and a model with a finite context window cannot read your whole site to find the three pages that matter. So let the site owner say which pages matter, in a single hand-writable text file.

The design borrows deliberately from robots.txt: one well-known location, plain text, simple enough to write by hand in ten minutes, no tooling required. Where robots.txt tells crawlers what they may read, llms.txt tells models what they should read. The proposal also introduced two companion conventions — Markdown versions of individual pages, and an expanded llms-full.txt file we cover below.

It is no accident that the earliest and densest adoption came from developer documentation. Docs are the clearest case where the incentives all point one way: if an AI coding assistant reads your API docs correctly, your users integrate faster and your support load drops. There is no scenario where a model understanding your docs better hurts you. Marketing sites have a murkier calculus, which is exactly why the "should you bother" section of this guide exists.

Here is the part most llms.txt guides bury, hedge, or omit: no major AI engine has formally committed to reading llms.txt. Not for retrieval, not for grounding, not for training. The state of play, engine by engine:

The Google position deserves the most attention, because it is the most explicit. Google has said that AI Overviews does not use llms.txt, and John Mueller has publicly compared the file to the keywords meta tag — the canonical example of a self-declared signal that engines stopped trusting because site owners say whatever flatters them. That comparison is pointed, and it is the strongest argument against expecting llms.txt to ever become a ranking or citation input: signals that sites control unilaterally have a poor track record of being consumed at face value.

On the other side of the ledger: Anthropic publishes an llms.txt for its own documentation at docs.anthropic.com — which tells you a major AI lab considers the format worth implementing for its own content, while committing to nothing about how its models treat yours. The tooling ecosystem is genuinely growing: documentation platforms generate llms.txt files automatically, community directories track sites that publish one, and AI coding agents will fetch and use one when a developer points them at a site. And in our experience, AI crawlers do show up in server logs requesting /llms.txt — though fetching a file is not the same as using it, and nobody outside those companies knows what happens after the fetch.

So the honest summary is asymmetric: consumption by the big engines is unproven and in Google's case denied; consumption by the long tail of tools and agents is real and growing. That asymmetry — not the hype — is what should drive your decision, and we will come back to it in the decision framework at the end.

The three root-level text files get conflated constantly, and the confusion produces real mistakes — most commonly, site owners assuming llms.txt controls AI access to their content. It does not. The three files answer three different questions:

robots.txt is permission. It tells crawlers which paths they may and may not fetch, per user agent. It is the only one of the three with any enforcement semantics, and it is the file you use if you want to allow or block AI crawlers — OpenAI, Anthropic, Perplexity, and Google all document their crawler user agents and state that they respect robots.txt directives. sitemap.xml is inventory. It enumerates every URL you want indexed, exhaustively and without commentary — its value is completeness, the opposite of curation. llms.txt is curation. It says, in human-readable language, "of everything on this site, these are the pages that matter, and here is what each one contains."

The other axis of difference is maturity. robots.txt has been honored since 1994 and was eventually formalized as an RFC. Sitemaps have been a cross-engine standard since 2005. llms.txt is a 2024 proposal with unconfirmed consumption. That gap matters operationally: you can build processes on the first two and trust they will keep working; llms.txt you ship as a low-cost bet, not a load-bearing dependency.

One contradiction worth flagging because we see it in the wild: publishing a carefully curated llms.txt while simultaneously blocking every AI crawler in robots.txt. Decide your actual policy first. If you want AI engines reading and citing your content — the working premise of GEO — your robots.txt, your sitemap, and your llms.txt should all agree on it.

The spec is short by design — the entire format fits in five rules, in this order. The hero mockup at the top of this page shows a complete, compliant file; here is what each part does:

1. An H1 with the site or project name. # Acme Analytics — this is the only required element in the whole spec. A file containing nothing but a name heading is technically valid.
2. A blockquote summary. > Acme Analytics is product analytics for B2B SaaS teams... — one short paragraph describing what the site is and what a reader will find. These are the most important sentences in the file: a model that reads nothing else should leave with an accurate one-line understanding of who you are. Write it like the first sentence of an encyclopedia entry, not a slogan.
3. Optional free-form prose. Plain paragraphs or lists, before any section headings, for context that does not fit a link list — version notes, how the docs are organized, what to read first.
4. H2-delimited sections of link lists. Each section is an H2 heading (## Docs, ## Guides, ## Policies) followed by Markdown list items in the form - [Page title](url): one-line description. The description after the colon is optional in the spec and essential in practice — it is what lets a model decide which link to follow without fetching all of them.
5. A section literally named "Optional". This name carries special semantics: links under ## Optional are the ones a model can safely skip when its context budget is short. It is the spec's pressure valve — it lets you include secondary content without diluting the primary list.

Serving details: the file must be plain text — served as text/plain or text/markdown, UTF-8 encoded, at the root path /llms.txt. Not an HTML page that renders Markdown, not a PDF, not behind a login. The companion convention from the same proposal: offer clean Markdown versions of your key pages (commonly by appending .md to the URL), so the links in your llms.txt can point a model at content with no HTML overhead at all. The companion files are optional; if your pages render readable HTML, linking to the normal URLs is fine.

The same proposal defines a second file: llms-full.txt. Where llms.txt is an index — links plus descriptions — llms-full.txt inlines the complete content of every listed page into one large Markdown document. Instead of telling a model where your documentation lives, it hands over the documentation itself in a single fetch.

The use case is genuine: a developer can paste one file into a chat session and give the model the entire corpus; an agent can grab everything it needs in one request instead of crawling fifty pages. For documentation sites with a finite, well-structured body of content, llms-full.txt is the more useful of the two files in day-to-day practice — it is the one developers actually reach for when they want an AI assistant to know a product's API.

The tradeoffs scale with your content. A full-corpus file gets large fast, and a file that exceeds the context window of the models reading it defeats its own purpose — past a point, you are back to needing an index. It is also a second artifact to keep fresh: an llms-full.txt that drifts out of date is worse than not shipping one, because it confidently feeds models stale content. And to pre-empt a common worry — publishing your content twice does not create a duplicate-content problem in search; the file is plain text that you would not want indexed as a page anyway. Keep your canonical HTML pages as the source of truth and treat llms-full.txt as a generated export, ideally produced by your docs tooling on every deploy rather than maintained by hand.

Our recommendation: docs-heavy sites should consider shipping both files, generated automatically. Marketing sites almost never need llms-full.txt — a curated llms.txt covers the realistic use cases.

Writing a good llms.txt takes about an hour, and most of that hour is editorial, not technical. The steps:

1. Inventory your candidate content. Pull the pages that genuinely explain who you are and what you do: documentation, integration guides, pricing, API references, core product pages, policies that customers ask about. Your sitemap and analytics are the starting list, not the final one.
2. Curate ruthlessly. A good llms.txt has on the order of ten to fifty links, not five thousand. The file's entire value is the judgment it encodes — if everything is important, nothing is, and you have rebuilt your sitemap in a worse format. For every page ask: "if a model could read only twenty pages before answering a question about us, is this one of them?"
3. Write the H1 and summary blockquote. Name, then two or three sentences that state what the company or site is in plain, answer-ready language. No superlatives — a model paraphrasing "the world's most revolutionary platform" extracts nothing usable. State category, audience, and what the linked content covers.
4. Group links into H2 sections that mirror how people use you. "Docs", "Guides", "Pricing", "Company" beat clever names. A model scanning section headings should be able to route a question to the right group instantly.
5. Write a one-line description for every link. This is where the file earns its keep. "API reference" tells a model little; "REST endpoints, authentication, and rate limits" tells it whether to follow the link. Write each description for a reader with zero context about your product.
6. Add the Optional section honestly. Changelog, older guides, secondary material — content that helps when context is plentiful and can be skipped when it is not.
7. Ship it to the root. Save as llms.txt, UTF-8, served at yourdomain.com/llms.txt with a plain-text content type. On most platforms this means dropping the file in your public/static directory. Single-page apps need care here — see common mistakes below.
8. Put it on a maintenance schedule. Add "update llms.txt" to your release checklist or review it quarterly. Every link in it should resolve, without redirects, forever or until edited.

On tooling: documentation platforms and CMS plugins increasingly generate llms.txt automatically, and for docs sites generated-on-deploy is the right answer. For everything else, hand-writing is not a fallback — it is the point. The curation is the value, and curation is a human judgment call.

Read enough published llms.txt files and clear patterns separate the useful ones from the noise. The strongest examples in the wild today are overwhelmingly developer-documentation sites — Anthropic, for instance, publishes an llms.txt for its docs at docs.anthropic.com — and they share a recognizable shape:

• Short and current. A screenful or two of links, every one of which resolves. The file reads like a well-maintained README, not an export.
• A summary that answers "what is this?" in two sentences. Factual, specific, paraphrasable. The test: if a model quoted only your blockquote, would the answer be accurate and useful?
• Descriptions written for strangers. Each link's one-liner assumes no prior knowledge — no internal jargon, no product codenames, no "learn more about our solutions."
• Sections that mirror the reader's journey. Getting started before reference, core concepts before edge cases. The structure itself teaches a model how the content fits together.
• Stable, canonical URLs. No tracking parameters, no redirects, no URLs that change with every site reorganization. The file points at permanent addresses.
• An Optional section that is actually optional. The discipline of admitting some of your content is secondary is itself a quality signal — files where everything is top-priority were not curated.

The weak ones share a shape too: machine-dumped sitemaps renamed to llms.txt, hundreds of undifferentiated URLs with no descriptions, marketing boilerplate in the summary, and dead links from a site migration nobody propagated. A model — or a human — learns nothing from them except that the file was shipped to tick a box. If your llms.txt would embarrass you as a public README, it is not done.

The failure modes are consistent enough to list. In rough order of how often we see them:

• Serving HTML instead of text. The single most common technical failure. Single-page apps routinely return the app shell — your homepage HTML — for any unknown path, including /llms.txt. The file "exists" in your repo and every fetcher on earth gets JavaScript boot code instead. Verify the live response, not the source tree.
• Dumping the sitemap. Five thousand links with no descriptions is an inventory, and you already have one at /sitemap.xml. Curation is the entire reason this file exists; skip it and you have shipped nothing.
• Writing the summary as ad copy. "The leading AI-powered revolutionary platform trusted by industry innovators" gives a language model zero extractable facts. State what you are, for whom, in plain declarative sentences.
• Broken and redirecting links. Every dead link teaches a consumer of the file to trust it less. Redirect chains waste the exact context budget the file exists to protect.
• Treating it as access control. llms.txt cannot block, allow, or rate-limit anything. If your goal is keeping AI crawlers out — or letting them in — that policy lives in robots.txt, and the two files should agree.
• Letting it rot. A stale file with dead links and renamed products actively misinforms any system that does read it. If you cannot commit to maintenance, a minimal five-link file you keep current beats an ambitious one you abandon.
• Expecting a ranking boost — and measuring for one. Google has said AI Overviews does not use the file. Shipping llms.txt and then attributing citation or traffic changes to it is reading signal into noise. Measure your AI visibility properly, across engines, against the content and authority work that actually moves it.

Testing llms.txt takes minutes and catches the failures that matter. The manual pass first:

1. Fetch it raw. Open yourdomain.com/llms.txt in a browser or with curl. You should see your Markdown as plain text — if you see your homepage, rendered HTML, or a 404, the file is not actually being served.
2. Check every link. Each one should return 200, directly, with no redirect hops. Do this again after any site migration.
3. Run the comprehension test. Paste the file into any AI chat and ask: "Based on this file, what does this company do, and where would you look for pricing and API documentation?" If the answers come back wrong or vague, the summary and descriptions need work. This is the cheapest llms.txt quality test that exists, and almost nobody runs it.

For the automated version: our free AI visibility checker scans robots.txt, sitemap.xml, and llms.txt on any homepage instantly — no signup, no email gate. It catches the serving failures (missing file, HTML response) and shows you all three files' status in one view, which matters because the three need to agree with each other.

One-off checks catch today's problems; sites drift. Inside the platform, Aergos Technical SEO crawls check robots.txt, sitemap.xml, llms.txt, and Core Web Vitals continuously and roll them into an AI-search readiness score — so when a deploy quietly breaks the file's content type or a migration kills half its links, you find out from a dashboard instead of from silence.

Strip away both the hype and the dismissals and the decision is straightforward, because the costs and the stakes are both small. The honest case for shipping one:

• The cost is near zero. An hour to write, minutes a quarter to maintain. There is no technical risk, no ranking risk, and nothing to undo if the standard dies.
• The upside is possible, not promised. If any major engine starts consuming llms.txt, sites with a good one are positioned on day one. If none ever does, you lost an hour.
• The long tail already reads it. AI coding agents, documentation tooling, and developers pasting your file into a chat session are real consumers today — small audiences, but high-intent ones, especially for developer products.
• The exercise has value regardless of consumption. Writing an llms.txt forces you to decide your canonical content map: which pages define you, what each one is for, in what order a stranger should read them. That judgment is the same raw material that answer-first content and citation-earning work are built from. Teams are routinely surprised by what the exercise surfaces — orphaned key pages, three URLs competing to explain the same thing, no page at all for the question customers ask most.

And the honest case against — or at least for deprioritizing it: if you run a five-page brochure site with no documentation and no content programme, the realistic consumers of your llms.txt round to zero, and the hour is better spent on a page that answers a real customer question. If you cannot commit to keeping the file current, skip it — a rotting file is worse than none. And if anyone on your team expects llms.txt to move citations or rankings on its own, reset that expectation before shipping: Google has said its AI surfaces do not use it, and no other engine has committed to it.

Keep the proportions straight. llms.txt is one small technical signal at the bottom of the AI-visibility stack. What actually earns citations across ChatGPT, Perplexity, Gemini, Claude, and Google AI Overviews is the content and authority work above it — being the source worth citing. Ship the file, then spend your real effort there.