llms.txt: definition, format, example and real limits

The llms.txt file is a Markdown index, placed at the root of a site, that describes the site and lists its reference pages for language models. This guide covers its definition, format, a complete example and its real limits. It is a proposal, not a standard: no body has standardised it and it has no directive value, unlike the robots.txt file that controls AI crawler access. Here is who reads it, how to write it in ten minutes, and why it comes after structured content, structured data and robots.txt.

llms.txt at a Glance: Definition, Origin, Status

A Proposal by Jeremy Howard, Published in September 2024

llms.txt was proposed in September 2024 by Jeremy Howard (Answer.AI), on the llmstxt.org site. The idea starts from a constraint of language models: they struggle to read HTML pages loaded with menus, scripts and interface blocks, and their context window cannot absorb an entire site. A short Markdown file at a fixed address (/llms.txt) gives them a summary of the site and the list of pages to read first.

Four Key Takeaways

Before getting into the format, four points frame the topic at the time of writing:

• An index, not a directive: the file describes and lists; it does not block, allow or prioritise anything for a crawler.
• A proposal, not a standard: no standards body has adopted it, and its form may still change.
• Reading is not guaranteed: no major search engine or LLM provider has confirmed reading it, and Google has stated that it does not use it.
• Near-zero cost: the file takes a few minutes to write and carries no risk as long as it stays consistent with the site.

Who Reads llms.txt Today, and Who Does Not

The decisive question is who reads it. The table below summarises each player's known position at the time of writing, and what it means for you.

Check Your Server Logs

The only way to know whether your file is being fetched is to look at your server logs. A standard web analytics tool sees nothing, because crawlers do not execute tracking scripts. Focus the analysis on a few elements:

• requests for /llms.txt and /llms-full.txt, with their response code;
• the user agents behind these requests: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot or developer tools;
• how often they visit, compared with the same crawlers on your HTML pages;
• requests for the .md versions of your pages, if you publish them.

A request proves that a crawler downloaded the file, not that a model uses it to answer.

Why Publish It Anyway

The lack of confirmed reading by the major players does not rule out publishing the file. Four reasons argue for doing it:

• the cost is near zero: a static text file of a few dozen lines;
• it serves agents and assistants that load it explicitly, particularly on technical documentation;
• it forces you to designate your reference pages, an editorial exercise that is useful well beyond the file;
• it carries no risk as long as it stays consistent with robots.txt, your canonical pages and your noindex tags.

The Format Set Out in the Proposal

The proposal describes a Markdown file with a simple structure. Only one element is mandatory; the others organise the reading. The table below lists each element in file order, followed by the two optional complements.

Title, Summary and Link Sections

The H1 title gives the site name, without a slogan. The blockquote that follows summarises in one sentence what the company does and for whom: it is the line a model will pick up most easily. Free-text paragraphs add context: languages, scope of the offer. Each H2 section then groups links in the form - [title](url): description, the description stating in a few words what the page provides.

The Optional Section, llms-full.txt and .md Versions

The ## Optional section flags the links a tool can skip when its context window is short: blog, archives, release notes. The /llms-full.txt file concatenates the full content of the reference pages in Markdown; it mainly suits large technical documentation. The .md versions of pages (page.html.md) offer the same content without menus or scripts. For a company website without technical documentation, the main file is enough.

Complete llms.txt File Example

Here is a complete file for a fictional B2B company, Optimalis, which publishes industrial maintenance software, on the fictional domain example.com. Each line of the list corresponds to one line of the file; the blank lines separating blocks are not reproduced.

• # Optimalis
• > Maintenance planning software for mid-sized industrial production sites.
• Optimalis publishes SaaS software available in French and English. The links below lead to the site's reference pages.
• ## Product
• - [Software overview](https://www.example.com/software): features, scope and integrations
• - [Pricing](https://www.example.com/pricing): current plans and terms
• - [Security and hosting](https://www.example.com/security): hosting, certifications, backups
• ## Documentation
• - [Getting started guide](https://www.example.com/docs/getting-started.html.md): installation and initial setup
• - [API reference](https://www.example.com/docs/api.html.md): endpoints and data formats
• - [Glossary](https://www.example.com/glossary): definitions of maintenance terms
• ## Company
• - [About](https://www.example.com/about): history, team, locations
• - [Contact](https://www.example.com/contact): sales and support contact details
• ## Optional
• - [Blog](https://www.example.com/blog): in-depth articles on industrial maintenance
• - [Release notes](https://www.example.com/release-notes): software update history

What This Example Shows

The file has fewer than twenty lines and contains only canonical pages that are accessible to crawlers and indexable. The descriptions state what each page provides, with no sales language. The documentation links point to .md versions, which the publisher serves alongside its HTML pages. The Optional section groups what a tool in a hurry can leave aside.

Nothing in this file looks like a rule: no "cite first", no list of forbidden pages. Such a line would have no effect.

llms.txt, robots.txt, sitemap.xml, Structured Data: Who Does What

The four mechanisms complement each other but do not play the same role. The table below compares them on what matters when deciding the order of your workstreams.

Structured data describes each page to search engines in a format they actually read; llms.txt is only an optional site-wide summary of it.

Access Control Happens in robots.txt

Blocking or allowing an AI crawler is done in robots.txt, with its user agent: GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, Google-Extended for the use of your content by Google's models. Writing "Disallow" in llms.txt does nothing: that syntax belongs to robots.txt.

Keep the Four Mechanisms Consistent

The main risk lies in contradictions between files. Before publishing, check three points:

• no page listed in llms.txt is blocked in robots.txt;
• no listed page is noindex, redirected or non-canonical;
• reference pages also appear in the sitemap and carry up-to-date structured data.

Create and Publish the File

Once your reference pages are chosen, writing takes about ten minutes. Whatever your CMS, the principle is the same: a static file served at the root. Publishing requires checking a few technical points:

• Location: at the root of the host, at www.yourdomain.com/llms.txt.
• Status: a direct HTTP 200 response, with no 301 or 308 redirect.
• Format: served as text/plain or text/markdown, UTF-8 encoded.
• Content: canonical pages only, with no parameters or URL variants.
• Consistency: checked against robots.txt and noindex tags.
• Cache: your CDN purged after every update, so it does not serve an outdated version.
• Review: every link opened once, every description reread.

Generator or Plugin: Review What It Produces

An online generator or a plugin for your CMS produces a first file from your sitemap. The result tends to list everything: tag pages, archives, sales pages with no content, generic descriptions. Keep the structure, retain only the reference pages, rewrite every description and check that the plugin does not overwrite your corrections at the next update.

A Multilingual Site

A multilingual site publishes one file per site or per host, with a section per language if needed that lists only the canonical pages of each version. Consistency with your hreflang tags and your international SEO strategy takes precedence over the content of the file.

Keep It Up to Date

A file that points to deleted pages or outdated pricing works against the site. Review it whenever a reference page changes: new offer, URL refactor, CMS migration.

What llms.txt Does Not Do

llms.txt is credited with powers it does not have. At the time of writing, the file:

• gives no directive: no crawler reads its lines as rules to apply;
• blocks nothing, neither crawling, nor training, nor access to restricted content, which fall under robots.txt and authentication;
• has no proven effect on citations by AI: a site that publishes one and gets cited shows a correlation, never causation;
• cannot be customised per model: there is only one file, with no section specific to Claude, ChatGPT or Perplexity;
• has nothing to do with MCP, which connects agents to tools and data sources.

What Really Matters to Get Cited

Generative engines cite pages, not indexes. What counts is content structured with H1, H2 and H3 headings, lists and tables, explicitly named entities, information that is consistent from one page to the next, and pages reachable by AI crawlers. That is the core of generative engine optimisation (GEO).

Page structure correlates with citation: pages with an H1-H2-H3 hierarchy are 2.8× more likely to be cited by AI (State of AI Search, 2025). This is a correlation about structure, not an effect of llms.txt; you will find other benchmarks in our GEO statistics.

Why llms.txt Comes After Everything Else

Priorities follow the readers. More than 50% of Google searches display an AI Overview (Squid Impact, 2025). Yet Google has stated that it does not read llms.txt to build these answers. Your efforts therefore go first to what they do read: content, structured data, robots.txt and sitemap. llms.txt comes next.

Common Mistakes

The most frequent mistakes come from confusion with robots.txt or from a file published and then forgotten. The table matches each one with its effect and its fix.

Restricted Content

Leaving a paid page out of llms.txt does nothing to protect it. Since the file is public, do not include internal URLs, pre-production environments or exports: protection comes from authentication and, for AI crawlers, from robots.txt.

Launch Checklist

Before considering the file published, go through these points:

• reference pages chosen, one canonical URL per topic;
• H1 title and summary blockquote written;
• factual link descriptions, with no sales language;
• Optional section reserved for secondary links;
• no rules or directives in the file;
• consistency checked with robots.txt, the sitemap and noindex tags;
• HTTP 200 response on /llms.txt, with no redirect, in UTF-8;
• CDN cache purged after publishing;
• server logs monitored to spot requests for the file;
• date of the next review set.

Depending on Your Situation

What comes next depends on where you stand:

• you are starting out in GEO: see what changes when you move from SEO to GEO;
• your reference pages lack substance: start by building an editorial strategy;
• you want to track crawling and indexing by Google: rely on Google Search Console;
• you want to know what AI engines take from your markup: read structured data and GEO;
• you want to measure your presence in AI answers: run a GEO audit;
• your customers work with agents: read our guide to AI agents.

To check everything at once, the GEO & SEO 360 audit in the Incremys platform verifies that AI crawlers can access your reference pages and that robots.txt, sitemap and llms.txt are consistent, and tracks your presence in AI answers keyword by keyword.

FAQ

Is llms.txt Mandatory?

No. No rule, search engine or LLM provider requires llms.txt at the time of writing. It is a proposal published in September 2024, with no standard status. Not having one does not penalise a site. Having one costs almost nothing and helps the tools that load it, provided the file stays consistent with your pages.

Does Google Read llms.txt?

No, at the time of writing. Google has publicly stated that it does not use llms.txt, neither for its search engine nor for AI Overviews and AI Mode. To be visible in Google, work on the quality and structure of your pages, your structured data, your robots.txt and your sitemap. llms.txt changes nothing in these results.

Does llms.txt Replace robots.txt?

No. robots.txt gives crawl directives that serious crawlers respect; llms.txt is an index of reference pages, with no directive value. The two files coexist at the root of the site. The first decides what crawlers may explore; the second merely signals the pages most worth reading.

Can You Block an AI Crawler With llms.txt?

No. llms.txt blocks nothing. To stop GPTBot, ClaudeBot, PerplexityBot or another AI crawler from exploring all or part of the site, you need a directive in robots.txt, with the relevant user agent. For restricted content, authentication remains the only real protection. A "Disallow" line written in llms.txt has no effect.

What Is the Difference Between llms.txt and llms-full.txt?

llms.txt is a short index: a title, a summary and annotated links to reference pages. llms-full.txt, an optional complement, contains the full text of those pages, concatenated in Markdown. The second mainly suits large technical documentation; a company website without that kind of documentation can make do with the first.

How Do I Know Whether My llms.txt Is Being Read?

Check your server logs. Filter requests for /llms.txt, then look at the user agents behind them and how often they come. A standard web analytics tool does not see them, because crawlers do not execute its scripts. A request only proves that the file was downloaded, not that a model uses it to build its answers.

Do You Need One llms.txt per Language?

No. One file per site is enough, or one per host if each language has its own domain or subdomain. On a multilingual site with a single host, create a section per language in the same file and list only the canonical pages of each version, consistent with your hreflang tags.