It is a proposed plain text file at the root of a site that points AI systems at a curated, clean version of your most important content, in the same spirit as robots.txt or sitemap.xml.

Do AI crawlers actually read llms.txt?

Adoption is partial and changing. Treat it as cheap insurance rather than a lever with a measurable return, and do not let it displace work on the pages themselves.

Is llms.txt the same as robots.txt?

No. robots.txt controls what crawlers are allowed to fetch. llms.txt is about pointing them at the content you most want used, and it grants no permissions on its own.

Answer engine optimization

llms.txt explained, and whether it does anything yet

What the llms.txt proposal is, which crawlers respect it today, and an honest read on whether adding one is worth your afternoon.

September 2, 2026 · 6 min read

llms.txt shows up on a lot of AEO checklists, usually presented as a straightforward win. It's cheap to add and it might help. It's also frequently oversold, and it's worth knowing what it does and doesn't do before you spend an afternoon on it.

What it is

llms.txt is a proposed convention: a markdown file at the root of your site, at /llms.txt, that points AI systems at a curated version of your most important content.

The idea is that a language model fetching your site has to deal with navigation, cookie banners, marketing sections, and markup that carries no meaning. A curated file skips all of that and says: here is what this site is, here are the pages that matter, here is a one-line description of each.

The format is deliberately simple. A heading with the site name, an optional summary, and then sections of links with short descriptions.

# Example Product

> One sentence on what this is.

## Docs
- [Getting started](https://example.com/docs/start): How to install and run it.
- [API reference](https://example.com/docs/api): Every endpoint and parameter.

## Company
- [Pricing](https://example.com/pricing): Plans and what each includes.

There's a companion idea, llms-full.txt, which contains the actual content in markdown rather than links to it.

What it is not

Three clarifications, because these get muddled constantly.

It is not robots.txt. robots.txt controls what crawlers are permitted to fetch, and it's a long-standing convention that crawlers broadly respect. llms.txt makes no permission claims at all. Adding one does not stop anything from being crawled, and it does not grant access to anything.

It is not a ranking signal. No search engine has said it uses llms.txt for ranking, and there's no mechanism by which it would be one.

It is not required for AI systems to read your site. They fetch your normal pages. llms.txt is a convenience, not a gate.

Does anything actually read it?

This is the honest part, and the answer is: adoption is partial, changing, and not well documented by the companies involved.

The convention was proposed in 2024 and has been picked up by a number of documentation platforms and developer tools, particularly for API docs, where it genuinely helps. Uptake among the major AI companies as a consumed standard has been slower and less clearly stated than most articles about it imply.

What that means practically: nobody can currently tell you a reliable measured return on adding one. Anyone presenting a case study proving lift from llms.txt alone should be read skeptically, because isolating that variable is very hard.

So should you add one?

Yes, with a proportionate amount of effort.

The case for it is that it costs very little, it's a static file, it carries no risk, and if adoption grows you're already there. For documentation-heavy products in particular, the format lines up well with how developers use AI tools to work with an API.

The case against spending real time on it is that it's insurance rather than a lever, and effort spent on it is effort not spent on the pages themselves, which are definitely being read.

A sensible split: give llms.txt an hour, and give the content of your actual pages the rest of the day.

If you write one, write it well

The common mistake is dumping a sitemap into it. A list of every URL with no descriptions is worse than nothing, because it's just a sitemap in a worse format.

What makes it useful:

Curate ruthlessly. Twenty genuinely important pages, not two hundred.
Write real descriptions. One sentence per link explaining what's on the page and when someone would need it. These descriptions are most of the value.
Lead with a clear summary. One or two sentences on what the product is, in plain language, naming it explicitly.
Group into sections. Docs, guides, company, and so on.
Keep it current. A stale file pointing at moved pages is actively unhelpful.

The thing that matters more

If you only have time for one of these, skip llms.txt and fix your pages.

An AI system fetching your site is reading the same HTML a person's browser gets. If your pricing page states a number, that number is available to it. If your homepage says you offer flexible plans for teams of every size, there's nothing there to quote, and no auxiliary file changes that.

llms.txt is a pointer. It's only worth having if the things it points at are worth reading. Get those right first, then add the pointer.

Frequently asked questions

What is llms.txt?: It is a proposed plain text file at the root of a site that points AI systems at a curated, clean version of your most important content, in the same spirit as robots.txt or sitemap.xml.
Do AI crawlers actually read llms.txt?: Adoption is partial and changing. Treat it as cheap insurance rather than a lever with a measurable return, and do not let it displace work on the pages themselves.
Is llms.txt the same as robots.txt?: No. robots.txt controls what crawlers are allowed to fetch. llms.txt is about pointing them at the content you most want used, and it grants no permissions on its own.