Write llms.txt and decide which AI crawlers get in.

What is llms.txt

llms.txt is a markdown file you publish on your own site so that a language model can find out what the site contains without crawling it first. It is a curated map, not a copy: a name, a short summary and a set of links, each with one line saying what an agent will get if it follows it. The current specification is version 2 of the proposal at llmstxt.org.

The format is markdown with a fixed order. The H1 with the project or site name is the only required element. After it comes an optional blockquote summary, an optional details paragraph, and then any number of H2 sections, each holding a markdown list of links:

# Example Company

> Example Company publishes documentation, SDKs and a changelog for the Example API.

Optional details go here: how to read the links, what is not listed, anything an agent should know before it fetches a page.

## Docs

- [Example API quick start](https://example.com/docs/quickstart): Install the client and make a first request.
- [Example API reference](https://example.com/docs/reference): Every endpoint, parameter, rate limit and error code.

## Optional

- [Example changelog](https://example.com/changelog): Dated release notes; secondary detail an agent can skip.

Three things the specification asks for, in its own words: concise, clear language; a brief informative description on each link; and avoid ambiguous terms or unexplained jargon. It also expects the file itself to stay small enough to fit in a context window while the detail sits behind the links, fetched only when it is needed. Its own test for a finished file is blunt: ask an agent a question about your site, give it only your llms.txt as a starting point, and see whether it finds the right page.

Two clarifications that matter in 2026. First, the proposal says the information is used on demand, when an agent needs information about a topic while assisting a user, and that its authors expected it to be useful mainly for inference rather than training. Second, there is no llms-full.txt any more: version 1 described a tool for expanding the file into a full context, and version 2 removed that context-expansion tooling, which also removed the special meaning the old Optional section carried. Optional sections are still allowed as a convention for links an agent can skip. Both points are from the v2 change log.

Nothing in the specification promises a ranking benefit or an automatic upload. The file is a courtesy to agents that are already reading your site, and it costs one text file to publish.

Where the file goes, and why /llms.txt

The file is named llms.txt and lives at /llms.txt on your host. The specification ties that name to two conventions you already have: /robots.txt and /sitemap.xml sit at the root for the same reason, and index.html gives every path a conventional place for its human-readable entry point. An llms.txt file is the LLM-readable version of that idea.

The path is not decoration. A file placed at any subpath, such as /docs/llms.txt, covers every URL under that path, and where more than one file applies an agent should use the most specific one. That is the rule that lets a project published to a subdirectory on a shared host take part. The specification considered and rejected the /.well-known/ prefix of RFC 8615 for exactly that reason: well-known URIs exist only at the origin root, and anyone who controls only a path can never add one.

Version 2 also adds a way for a page to point at its markdown version and at the file that covers it, using the standard link relations rel="alternate" type="text/markdown" and rel="describedby", as HTML link elements or as an HTTP Link header. If you want an agent to find the file and the markdown versions of your pages without guessing, those two relations are worth adding to your head.

Check yours the same way you check anything else: fetch the URL in a browser and confirm it returns 200 and plain text rather than a redirect to a page. This site publishes its own file at /llms.txt if you want a worked example of the finished format.

How to check whether a crawler is allowed

Four checks, from cheapest to most convincing.

• Read the file itself. Open https://yourdomain.com/robots.txt in a browser. A 200 and plain text means the file is being served, which is the failure people hit most often: a redirect, a 404, or an HTML error page all look like a missing file to a crawler.
• Find the group, then read it. Locate the User-agent: line for the exact token. Matching is case-insensitive on the token the crawler sends. If the group has only Allow: / the crawler may fetch everything; Disallow: / blocks everything; Disallow: /path blocks one area.
• Resolve overlaps by hand. When more than one rule matches a URL, the longest matching path wins, and if two rules match with the same length, Allow wins. That rule is what makes Disallow: /admin plus Allow: /admin/public behave the way you expect.
• Watch a real fetch. Server logs are the ground truth for your own traffic, and a console crawling tool from a vendor will show you what its crawler is actually allowed to reach. Check the log after the deploy rather than trusting the file alone.

Two cautions. First, robots.txt is a request: RFC 9309 states that these rules are not a form of access authorization, so anything that must stay private needs authentication. Second, user-triggered fetchers are the least predictable group, because the vendors themselves say so. OpenAI states that robots.txt rules may not apply to ChatGPT-User, Perplexity states that Perplexity-User generally ignores robots.txt rules, and Meta states that Meta-ExternalFetcher may bypass robots.txt while sending the visitor IP address. A Disallow line tells those crawlers what you would prefer; it does not stop them.

Finally, do not expect a Crawl-delay line to be honoured. RFC 9309 defines no such field, and support for it is a vendor convention that some crawlers implement and others ignore. The generator above emits none.