llms.txt: the standard file that helps AI understand your website

Large language models are changing how people find information online. But there is a fundamental problem: LLMs can't read your entire website. Context windows are limited, HTML is messy, and converting pages with navigation, ads, and JavaScript into clean text is difficult.

llms.txt is an open specification designed to solve this. It is a simple markdown file at your site's root (/llms.txt) that provides LLM-friendly content: brief summaries, structured guidance, and links to detailed markdown files.

Think of it as a curated introduction to your site, specifically for AI assistants.

Why llms.txt exists

When an AI assistant like ChatGPT or Claude tries to understand your website, it faces several challenges:

1. Context window limits - Most LLMs can't process entire websites at once
2. HTML complexity - Navigation, ads, JavaScript, and styling add noise
3. No clear starting point - Which pages matter most for understanding your project?
4. Imprecise conversion - Turning HTML to plain text loses structure and context

llms.txt addresses these by providing:

• A single, accessible location for LLM-optimized content
• Clean markdown that both humans and models can read easily
• Structured metadata that classical parsers can handle (not just LLM inference)
• Links to detailed documentation in markdown format

The llms.txt specification

The format is deliberately simple. Here is what goes in /llms.txt:

Required sections

1. H1 heading - Your project or site name
2. Blockquote summary - Brief overview containing key information

Optional sections

3. Additional details - Paragraphs, lists, or other markdown (no headings)
4. File lists - H2-delimited sections with markdown lists of URLs

Each file list entry is formatted as:

- [Link title](url): Optional description

There is also an "Optional" section (H2: ## Optional) for secondary content that LLMs can skip when they need shorter context.

Example structure

# FastHTML

> FastHTML is a Python library combining Starlette, Uvicorn, HTMX, and fastcore's FT into a framework for server-rendered hypermedia applications.

Important notes:

- Not compatible with FastAPI syntax
- Works with vanilla JS and web components, not React/Vue/Svelte

## Docs

- [FastHTML quick start](https://docs.example.com/quickstart.html.md): Brief overview
- [HTMX reference](https://github.com/bigskysoftware/htmx/blob/master/www/content/reference.md): All HTMX attributes and methods

## Examples

- [Todo app walkthrough](https://github.com/example/app.py): Complete CRUD app showing FastHTML patterns

## Optional

- [Starlette documentation](https://example.com/starlette.md): Subset useful for FastHTML development

Markdown versions of pages (.md convention)

The specification also recommends providing markdown versions of your HTML pages by appending .md to URLs:

• Regular page: https://example.com/docs/api.html
• Markdown version: https://example.com/docs/api.html.md

For URLs without filenames, use index.html.md:

• Regular page: https://example.com/guides/
• Markdown version: https://example.com/guides/index.html.md

This gives LLMs access to clean, text-based versions without HTML overhead.

Who should use llms.txt

This specification is particularly valuable for:

Software documentation sites

If you maintain API docs, SDK guides, or technical tutorials, llms.txt helps AI assistants answer developer questions correctly. Example: "How do I authenticate with the API?"

Open-source projects

GitHub repositories with extensive READMEs, wikis, or doc sites benefit from structured llms.txt files that guide LLMs to installation guides, usage examples, and API references.

Educational content

Tutorial sites, course platforms, and knowledge bases can use llms.txt to surface key learning paths and reference materials.

E-commerce sites

Product catalogs, return policies, and FAQ pages can be surfaced through llms.txt so AI assistants give accurate information about your offerings.

Business websites

Company structure, services offered, team info, and case studies can be organized so LLMs understand your business accurately.

Real-world examples

Several projects have adopted llms.txt:

Anthropic Claude documentation

https://docs.anthropic.com/llms.txt
https://docs.anthropic.com/llms-full.txt

Anthropic's documentation follows the spec, providing both concise and comprehensive versions.

FastHTML project

https://fastht.ml/docs/llms.txt

The FastHTML team uses llms.txt to guide LLMs through their Python framework documentation.

nbdev projects

All Answer.AI and fast.ai projects using nbdev now generate .md versions of documentation pages automatically.

How to create an effective llms.txt file

1. Start with the essentials

Write a clear H1 and blockquote that answer:

• What is this project/site?
• What is its primary purpose?
• What should someone know upfront?

2. Organize by priority

Use H2 sections to group related content:

• Getting Started - Installation, quickstart guides
• Core Concepts - Key ideas users must understand
• API Reference - Detailed technical docs
• Examples - Real-world usage patterns
• Optional - Advanced topics, historical context, deep dives

3. Write concise link descriptions

Each link should have a brief, informative note:

- [User authentication guide](https://example.com/auth.html.md): OAuth2 setup with code examples

Not:

- [Guide](https://example.com/auth.html.md): Authentication

4. Test with actual LLMs

Generate expanded context files and test whether models can:

• Answer basic questions about your project
• Find specific documentation
• Understand your API structure
• Explain core concepts accurately

5. Keep it updated

llms.txt should evolve with your project. When you add major features or documentation sections, update the file accordingly.

Expanded context files (llms-ctx.txt)

Some projects generate expanded versions that concatenate the linked content:

• llms-ctx.txt - Expanded content without "Optional" section URLs
• llms-ctx-full.txt - Expanded content including "Optional" URLs

These files are created by tools like llms_txt2ctx that parse the llms.txt file and fetch linked content. This is useful for:

• AI agents that need full context upfront
• Testing how well your documentation answers questions
• Providing precomputed context for specific use cases

The FastHTML project uses this approach, generating XML-structured context files from their llms.txt.

Current adoption and ecosystem

Adoption status

As of 2026, llms.txt adoption is in early stages:

• No official support from major AI providers (OpenAI, Anthropic, Google) has been announced
• Limited crawling - Server logs show minimal bot traffic specifically requesting llms.txt
• Growing awareness - Developer communities are discussing and implementing it
• Proactive projects are adding it as future-proofing

Community tools and integrations

Several tools support llms.txt:

• llms_txt2ctx - CLI and Python module for parsing and expanding llms.txt files
• JavaScript implementation - Sample JS parser for the spec
• VitePress plugin - Auto-generates llms.txt for VitePress documentation sites
• Docusaurus plugin - Generates LLM-friendly docs following the spec
• Drupal Recipe - Full llms.txt support for Drupal 10.3+ sites
• llms-txt-php - PHP library for reading and writing llms.txt files
• VS Code PagePilot Extension - Chat participant that loads llms.txt context automatically

Directories

Two directories catalog llms.txt files on the web:

• llmstxt.site
• directory.llmstxt.cloud

Criticism and trade-offs

The SEOPub critique

Some SEO professionals have raised concerns:

"They don't benefit your website at all. They only benefit the LLM providers. There is no reference to original URLs, so if an LLM cites you, the link goes to the .md file - just a wall of text. Horrible user experience."

This is a valid concern. The spec should be extended to:

• Ensure .md versions include canonical URL references
• Provide guidance on citation formatting
• Consider how LLM-generated citations link back to HTML pages, not markdown sources

The John Mueller perspective

Google's John Mueller compared llms.txt to the meta keywords tag:

"This is what a site-owner claims their site is about... At that point, why not just check the site directly?"

Fair point. LLMs could theoretically parse HTML directly. But llms.txt advocates argue:

• Context windows make full-site parsing impractical
• Clean markdown is easier to process than HTML
• Curated content reduces noise and improves accuracy
• Semantic structure helps models reason about content

The "too early" argument

Many developers note that it is premature to invest heavily in llms.txt since:

• No major AI service officially uses it
• The spec might evolve or be replaced
• Effort could be better spent on traditional SEO and user-facing docs

Counter-argument: early adopters shape emerging standards. If llms.txt gains traction, sites with existing files will benefit first.

How llms.txt compares to other standards

robots.txt

• Purpose: Control automated crawler access
• Audience: Search engine bots
• Use case: Prevent indexing of specific paths

sitemap.xml

• Purpose: List all indexable pages
• Audience: Search engine crawlers
• Use case: Help search engines discover and index content

llms.txt

• Purpose: Provide curated, LLM-friendly content
• Audience: Large language models (ChatGPT, Claude, Gemini)
• Use case: Help AI assistants understand your site during inference

Key difference: llms.txt is for inference time, not training or indexing. It helps when a user asks an AI assistant a question and the model needs to understand your site right now.

Should you add llms.txt to your site?

Yes, if you:

• Run a documentation site with frequent AI-related queries
• Want to future-proof your site for LLM-based discovery
• Have complex content that benefits from curation
• Can generate markdown versions of your pages easily
• Are experimenting with AI-native workflows

Maybe not, if you:

• Have a simple, single-page site
• Lack resources to maintain markdown versions
• Prefer to wait for official LLM provider support
• Focus on traditional search traffic

A balanced approach

Even if llms.txt is not widely adopted today, the practice of creating clean, structured markdown documentation is valuable on its own. You can:

1. Create a basic llms.txt file (low effort, future-proofing)
2. Generate markdown versions of key pages (useful for LLMs and human readers)
3. Monitor logs for llms.txt requests (track if AI services start using it)
4. Update the file as your docs evolve

Worst case: you have better-organized documentation. Best case: you are ready when LLMs standardize on this approach.

Creating your llms.txt file

Manual approach

1. Create /llms.txt in your site root
2. Write H1, blockquote summary, and optional details
3. Add H2 sections with file lists
4. Generate .md versions of key pages
5. Test the file by expanding it and asking LLMs questions

Automated tools

Use plugins or CLI tools:

• VitePress plugin for VitePress sites
• Docusaurus plugin for Docusaurus sites
• Drupal Recipe for Drupal sites
• Custom build scripts for static site generators

Generator tool

For a quick start, use the llms.txt generator on ExplainX - a free tool that creates properly formatted llms.txt files based on your project details.

The future of llms.txt

The specification is community-driven and open for input. A GitHub repository hosts the informal overview, and a Discord channel facilitates discussion.

For llms.txt to succeed, it needs:

1. LLM provider adoption - OpenAI, Anthropic, Google, and others officially supporting it
2. Tooling maturity - Better generators, validators, and analytics
3. Clear value proposition - Measurable impact on LLM response accuracy
4. Spec evolution - Address citation, attribution, and URL reference issues

Until then, it remains an experiment worth watching - and potentially participating in.

Related reading on ExplainX

• DESIGN.md: the open spec for AI design systems - Structured design intent for agents
• What are agent skills? Complete guide - Reusable agent workflows
• What is MCP? Model Context Protocol explained - How agents connect to tools and data

Bottom line

llms.txt is a simple, structured way to help AI assistants understand your website. While adoption is early and not all major LLM providers officially support it yet, the underlying idea is sound: give models clean, curated content so they can provide accurate answers about your project.

Whether llms.txt becomes the standard or not, creating well-organized, markdown-based documentation benefits both human readers and AI systems. If you are already maintaining docs, adding a llms.txt file is a low-effort experiment that could pay off as LLM-based discovery becomes more common.

The spec is open, the tools exist, and the worst-case outcome is slightly better documentation. That is a reasonable bet on the future of AI-assisted information retrieval.

Official specification: llmstxt.org

Community directory: llmstxt.site

FastHTML example: fastht.ml/docs/llms.txt

Anthropic example: docs.anthropic.com/llms.txt