llms.txt is a proposed standard for a Markdown file placed at the root of a website (at /llms.txt) that gives language models a short description of the site and a hand-picked list of links to its most useful content. It is meant to be read by AI tools when they need information about a site, so they can find clean text without parsing full HTML pages. It is a community proposal, and support from AI companies is limited and uneven.
What problem is llms.txt trying to solve?
A web page built for people carries a lot that a language model does not need: navigation menus, cookie banners, scripts, ads and footers. When an AI tool fetches a page to answer a question, it has to strip all of that away. Some content only appears after JavaScript runs, and many fetchers never run it.
Models also work with a limited context window, the amount of text they can consider at once, which is measured in tokens. A whole website rarely fits. llms.txt is a short index, written by the site owner, that says what the site is and which pages matter, in a format a model can read with little processing.
What does an llms.txt file contain?
The proposal defines a simple structure in Markdown, a plain-text format in which # marks a heading, > marks a quoted block and text marks a link. The parts appear in this order:
- An H1 heading with the name of the site or project. This is the only required part.
- A blockquote with a one- or two-sentence summary of what the site is.
- Optional paragraphs or lists with more context, such as how the product or documentation is organized.
- One or more H2 sections, each containing a list of links. Each link can be followed by a colon and a short note on what the page covers.
- An optional section titled "Optional", holding links a tool can skip when it is short on space.
A small file for a fictional company could look like this:
# Example Co
> Example Co makes accounting software for small bakeries. This file lists our main documentation and policy pages.
## Docs
- Getting started: installing the app and setting up an account
- Pricing: plans and what each one includes
## Optional
- Company historyThe proposal also suggests that sites publish a Markdown version of each important page at the same URL with .md added, so the links in llms.txt can point to clean text. Some sites go further and publish llms-full.txt, a single file that joins the full text of their documentation together.
How is llms.txt different from robots.txt and sitemaps?
All three are files at the root of a site that machines read, which makes them easy to confuse, but each does a different job.
| robots.txt | sitemap.xml | llms.txt | |
|---|---|---|---|
| Purpose | Tells crawlers which URLs they may fetch | Lists URLs for search engines to discover | Summarizes the site and points to its most useful content |
| Format | Plain-text directives | XML | Markdown |
| Scope | Rules for URL patterns across the site | Usually every indexable URL | A short, curated selection |
| Status | Established convention followed by compliant crawlers | Supported by major search engines | Proposal with limited adoption |
The biggest difference is that llms.txt grants or denies nothing. Listing a page in it does not give a bot permission to fetch that page, and leaving a page out does not block it. Access rules stay in robots.txt, where each of the AI crawlers can be allowed or blocked by its user agent name.
Do AI search engines use llms.txt?
No major AI search product has publicly committed to using llms.txt to decide which pages to retrieve or cite, and Google has said its search systems do not rely on it. AI search answers are built through retrieval-augmented generation: a search index returns candidate pages, a ranking step scores passages, and the model quotes the ones it uses. How AI search chooses which sources to cite describes those stages. They depend on the index and the page content, and none of them reads a site's llms.txt first.
The file is used in narrower settings. Coding assistants and developer tools load documentation into a model's context, often because a user pasted the llms.txt URL or configured the tool to fetch it. Several documentation platforms now generate the file automatically for the sites they host. Server logs on some sites show bots requesting /llms.txt, though a request only proves the file was fetched and says nothing about whether its content influenced an answer.
Should a website publish an llms.txt file?
The cost is small: one static text file. The benefit depends on who reads the site. A software company whose users work with AI coding tools has a clear use for it, because those tools do read such files when pointed at them. For a shop, a restaurant or a local service business, there is little evidence that the file changes anything today.
If you do publish one, a few practices keep it useful:
- Link only to public, canonical pages that load without errors.
- Write the summary as a plain description of what the site or product is, the same way you would explain it to a new customer.
- Update the file when pages move or get renamed, since hand-maintained link lists break without any warning.
- Leave out anything you would not want public. Anyone can open the file in a browser.
- Keep all crawl permissions in robots.txt, because llms.txt has no effect on them.
AI search platforms do not report whether they read the file. To judge its effect, compare how often the site is mentioned or cited in AI answers before and after publishing it, which is what tracking AI visibility over time records.
