llms.txt: what it is, what it does not do, what the evidence says

llms.txt is a markdown file at the root of a site that gives a language model a short entry point: the site's name, a summary and lists of links. This page says what the proposal contains, what it leaves out, where llms-full.txt came from, what the measurements show and what the sceptics say. Every statement carries its source and read date, and statements by vendors about themselves are labelled.

What the proposal says

• llms.txt is a markdown file at /llms.txt (or any subpath) meant to give LLMs a concise entry point to a site. [primary source] source, read 4 November 2026.
• Sections in order: optional byte-order mark, an H1 with the site or project name (the only required part), an optional blockquote summary, zero or more detail sections without headings, and zero or more H2 file lists. [primary source] source, read 4 November 2026.
• Each file-list entry is a markdown link [name](url), optionally followed by a colon and notes. [primary source] source, read 4 November 2026.
• An H2 named Optional holds secondary links that an agent can skip when it needs less context. [primary source] source, read 4 November 2026.
• Pages should have a markdown copy at the same URL as page.html.md or page.md, with index.html.md or index.md for URLs without a file name. [primary source] source, read 4 November 2026.
• Discovery uses rel="alternate" type="text/markdown" for page copies and rel="describedby" for the covering llms.txt, in HTML link elements or HTTP Link headers. [primary source] source, read 4 November 2026.
• The llmstxt.org site itself serves a 637-byte llms.txt with one H2 and three links (read 2026-11-04). [primary source] source, read 4 November 2026.

What it does not do

• llms.txt does not control crawling; robots.txt does. Fern describes robots.txt as deciding which crawlers reach a site and llms.txt as shaping what agents receive once they do. [primary source] source, read 4 November 2026.
• The proposal text read on 2026-11-04 does not define llms-full.txt. [primary source] source, read 4 November 2026.
• ReadMe tells customers whether an agent requests the file is up to the agent. [primary source] source, read 4 November 2026.
• Google Search says it does not need llms.txt and that the file will neither help nor hurt rankings. [primary source, 2026-06-15] source, read 4 November 2026.

llms-full.txt

• GitBook publishes /llms-full.txt with the full documentation content in one file, including hidden pages. [primary source] source, read 4 November 2026.
• Scalar llms-full.txt concatenates the complete markdown of pages in the same navigation order as llms.txt. [primary source] source, read 4 November 2026.
• Shopify serves /llms-full.txt as an alternate URL mirroring /agents.md. [primary source] source, read 4 November 2026.
• Mintlify exempts llms-full.txt from the 100,000-character split it applies to llms.txt. [primary source] source, read 4 November 2026.
• docusaurus-plugin-llms writes llms-full.txt by bundling all docs content into one document at build time. [primary source] source, read 4 November 2026.

History and versions

• v1 was published by Jeremy Howard on 2024-09-03. [primary source, 2024-09-03] source, read 4 November 2026.
• v2 is dated August 2026; the llmstxt.org page shows a modified date of 2026-08-10. [primary source, 2026-08-10] source, read 4 November 2026.
• v2 removed the llms_txt2ctx tool from the proposal and says agents view or search llms.txt and then follow relevant links. [primary source] source, read 4 November 2026.
• v2 defines subpath llms.txt files as covering pages under their path, with the most specific file applying. [primary source] source, read 4 November 2026.
• Yoast added llms.txt generation to its free WordPress plugin in version 25.3, announced 2025-06-02. [vendor claim, 2025-06-02] source, read 4 November 2026.
• Chrome Lighthouse added an llms.txt audit to its Agentic Browsing category; a 404 is marked not applicable. [primary source] source, read 4 November 2026.

The evidence on whether it matters

Ora ran one docs site in 19 configurations and probed it with a plain-fetch client and a JavaScript-running client on five topics (190 probes). Only two changes stopped retrieval, and neither was about llms.txt. The rest of the evidence comes from traced agent runs and from one site operator's logs.

• Ora's lab found (2026-10-19) that across 19 configurations of one docs site and 190 probes, llms.txt, JSON-LD, sitemaps and markdown mirrors made no difference to whether an answer already in the HTML was retrieved. [vendor claim, 2026-10-19] source, read 4 November 2026.
• Ora's lab found (2026-10-19) that 86 percent of llms.txt fetches arrived through a link, 93 percent for .well-known and 97 percent for openapi.json. [vendor claim, 2026-10-19] source, read 4 November 2026.
• In the Vercel and Ora data, agents requested markdown on 2,259 of 3,470 web fetches (65.1 percent), and chose markdown in 95.7 percent of fetches that named any format. [primary source, 2026-08-28] source, read 4 November 2026.
• Of 329 runs that reached llms.txt, 120 (36 percent) grounded the final answer in its content. [primary source] source, read 4 November 2026.
• Cloudflare Radar found markdown content negotiation passing on 3.9 percent of scanned top domains at the April 2026 launch of its agent readiness score. [primary source, 2026-04-17] source, read 4 November 2026.
• By default Cloudflare isitagentready.com checks markdown negotiation and leaves the llms.txt check opt-in. [primary source] source, read 4 November 2026.
• A site operator saw ChatGPT, Claude on the web and Perplexity fetch HTML over 13,000 times and markdown once while answering people (4 to 18 September 2026). [secondary source, 2026-09-22] source, read 4 November 2026.

Who generates it automatically

• Docs platforms that generate llms.txt by default: Mintlify, GitBook, ReadMe, Fern, Redocly and Scalar Docs. [primary source] source, read 4 November 2026.
• Wix auto-generates a default llms.txt that points agents to the Wix MCP endpoint. [primary source] source, read 4 November 2026.
• Shopify manages /llms.txt for every store as a mirror of /agents.md. [primary source] source, read 4 November 2026.
• WordPress gets it through plugins: Yoast SEO, All in One SEO, Website LLMs.txt. [primary source] source, read 4 November 2026.
• Webflow and Framer serve an owner-supplied file but do not write it. [primary source] source, read 4 November 2026.
• Firecrawl generates a one-off file from a crawl; its hosted endpoint has been unmaintained since 2025-06-30. [primary source] source, read 4 November 2026.

Arguments against

• Google Search says it does not use the file and that no special files are needed for AI Overviews or AI Mode. [primary source] source, read 4 November 2026.
• Scrunch does not use llms.txt in AXP and says consumer AI systems do not reliably read it. [vendor claim] source, read 4 November 2026.
• The Vercel and Ora guide says llms.txt, markdown mirrors and JSON-LD do not replace accurate, fetchable source pages, and agents can use llms.txt only after reaching the site. [primary source] source, read 4 November 2026.
• Large single files exceed context windows; Cloudflare saw agents fall into a grep loop on oversized llms.txt files. [primary source] source, read 4 November 2026.
• ReadMe calls llms.txt control over what an agent finds when it looks, not a guarantee that it will look. [primary source] source, read 4 November 2026.

The numbers in one place

Find a file and check yours

Three data pages extend the table above: what agents reach sets Ora's two studies side by side, llms.txt size measures the files and unlinked llms.txt lists sites that serve one and do not link it. llms.txt examples that score well on Ora joins the index to Ora's cached site scores.

The llms.txt Index renders 401 files with their parse and grade. The generator writes a file from a list of URLs. The guide to writing one that gets followed has the structure and the linking. To rate a whole site against the same read-layer behaviours, the check we recommend is Ora's scan, whose Access layer lists 42 checks (Ora's catalogue, read 5 November 2026). Cloudflare's scanner at isitagentready.com is a free quick alternative, and Mintlify's score grades documentation.

Sources

1. llms.txt proposal (llmstxt.org) https://llmstxt.org/, read 4 November 2026
2. llmstxt.org llms.txt https://llmstxt.org/llms.txt, read 4 November 2026
3. Fern docs: llms.txt https://buildwithfern.com/learn/docs/ai-features/llms-txt, read 4 November 2026
4. ReadMe docs: LLMs.txt https://docs.readme.com/main/docs/llmstxt, read 4 November 2026
5. Google Search Central documentation updates https://developers.google.com/search/updates, read 4 November 2026
6. GitBook docs: LLM-ready docs https://gitbook.com/docs/publishing-documentation/llm-ready-docs, read 4 November 2026
7. Scalar docs: llms.txt https://scalar.com/products/docs/configuration/llms-txt, read 4 November 2026
8. Shopify: agents.md.liquid https://shopify.dev/docs/storefronts/themes/architecture/templates/agents-md-liquid, read 4 November 2026
9. Mintlify docs: llms.txt https://www.mintlify.com/docs/ai/llmstxt, read 4 November 2026
10. docusaurus-plugin-llms https://github.com/rachfop/docusaurus-plugin-llms, read 4 November 2026
11. llms.txt v1 to v2 changes https://llmstxt.org/changes.html, read 4 November 2026
12. Yoast press release: llms.txt generator https://yoast.com/press/yoast-launches-the-first-llms-txt-generator-in-an-seo-plugin-ushering-in-a-new-era-of-ai-ready-content/, read 4 November 2026
13. Chrome Lighthouse: llms.txt audit https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt, read 4 November 2026
14. Ora: AgentReady launch post, findings 01 to 04 https://ora.ai/blog/agentready-launch, read 4 November 2026
15. Vercel and Ora: Make your site readable by AI agents https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents, read 4 November 2026
16. Cloudflare: agent readiness score and Radar numbers https://blog.cloudflare.com/agent-readiness/, read 4 November 2026
17. GoodBarber: Do AI agents read Markdown https://www.goodbarber.com/blog/do-ai-agents-read-markdown-what-two-weeks-of-our-own-server-logs-say-a1622/, read 4 November 2026
18. Wix Llms Txt API https://dev.wix.com/docs/api-reference/business-management/marketing/seo/txt-file-server/llms-txt/introduction, read 4 November 2026
19. Shopify: llms.txt.liquid https://shopify.dev/docs/storefronts/themes/architecture/templates/llms-txt-liquid, read 4 November 2026
20. Yoast llms.txt functional specification https://developer.yoast.com/features/llms-txt/functional-specification/, read 4 November 2026
21. Webflow help: Upload an llms.txt file https://help.webflow.com/hc/en-us/articles/43240104183315-Upload-an-llms-txt-file-to-your-site, read 4 November 2026
22. Firecrawl docs: LLMs.txt generator endpoint https://docs.firecrawl.dev/features/alpha/llmstxt, read 4 November 2026
23. Google Search Central: AI features https://developers.google.com/search/docs/appearance/ai-features, read 4 November 2026
24. Scrunch help: How AXP serves content to AI agents https://helpcenter.scrunchai.com/en/articles/12845899-how-axp-serves-content-to-ai-agents, read 4 November 2026

Last reviewed 4 November 2026