---
title: "llms.txt reference and the evidence: Agent Ready HTML"
description: "The llms.txt proposal, llms-full.txt, version 2, who generates the file and the measurements on whether agents use it. Ora's lab findings are dated."
canonical: https://agentreadyhtml.com/llms-txt/
site: "Agent Ready HTML"
last_updated: 2026-11-04
section: "Reference"
---

# llms.txt reference and the evidence: Agent Ready HTML

> The llms.txt proposal, llms-full.txt, version 2, who generates the file and the measurements on whether agents use it. Ora's lab findings are dated.

Source: https://agentreadyhtml.com/llms-txt/
Site: Agent Ready HTML
Last reviewed: 2026-11-04

# llms.txt: what it is, what it does not do, what the evidence says

In short

llms.txt is a markdown file proposed on 3 September 2024 that lists a site's best pages for a model. Version 2 dates from August 2026. In Ora's lab it changed nothing when the answer was already in the HTML, 86 percent of fetches arrived through a link, and Google says it does not use the file.

The check we recommend

Ora's scan. Its Access layer runs 42 checks worth 30 of its 100 points, from content without JavaScript to markdown on request. Every check is listed at [ora.ai/api/checks](https://ora.ai/api/checks?utm_source=agentreadyhtml.com&utm_medium=referral&utm_campaign=reference) with its tier, and 25 of the 132 with a spec link. Reads need no key and no sign-up (Ora's pricing page, read 5 November 2026).

Ora reads public URLs only, says two runs of one scan land within a couple of points without publishing repeat-scan data, and keeps the product that fixes what it finds behind a waitlist.

llms.txt is a markdown file at the root of a site that gives a language model a short entry point: the site's name, a summary and lists of links. This page says what the proposal contains, what it leaves out, where llms-full.txt came from, what the measurements show and what the sceptics say. Every statement carries its source and read date, and statements by vendors about themselves are labelled.

## What the proposal says

-   llms.txt is a markdown file at /llms.txt (or any subpath) meant to give LLMs a concise entry point to a site. \[primary source\] [source](https://llmstxt.org/), read 4 November 2026.
-   Sections in order: optional byte-order mark, an H1 with the site or project name (the only required part), an optional blockquote summary, zero or more detail sections without headings, and zero or more H2 file lists. \[primary source\] [source](https://llmstxt.org/), read 4 November 2026.
-   Each file-list entry is a markdown link \[name\](url), optionally followed by a colon and notes. \[primary source\] [source](https://llmstxt.org/), read 4 November 2026.
-   An H2 named Optional holds secondary links that an agent can skip when it needs less context. \[primary source\] [source](https://llmstxt.org/), read 4 November 2026.
-   Pages should have a markdown copy at the same URL as page.html.md or page.md, with index.html.md or index.md for URLs without a file name. \[primary source\] [source](https://llmstxt.org/), read 4 November 2026.
-   Discovery uses rel="alternate" type="text/markdown" for page copies and rel="describedby" for the covering llms.txt, in HTML link elements or HTTP Link headers. \[primary source\] [source](https://llmstxt.org/), read 4 November 2026.
-   The llmstxt.org site itself serves a 637-byte llms.txt with one H2 and three links (read 2026-11-04). \[primary source\] [source](https://llmstxt.org/llms.txt), read 4 November 2026.

## What it does not do

-   llms.txt does not control crawling; robots.txt does. Fern describes robots.txt as deciding which crawlers reach a site and llms.txt as shaping what agents receive once they do. \[primary source\] [source](https://buildwithfern.com/learn/docs/ai-features/llms-txt), read 4 November 2026.
-   The proposal text read on 2026-11-04 does not define llms-full.txt. \[primary source\] [source](https://llmstxt.org/), read 4 November 2026.
-   ReadMe tells customers whether an agent requests the file is up to the agent. \[primary source\] [source](https://docs.readme.com/main/docs/llmstxt), read 4 November 2026.
-   Google Search says it does not need llms.txt and that the file will neither help nor hurt rankings. \[primary source, 2026-06-15\] [source](https://developers.google.com/search/updates), read 4 November 2026.

## llms-full.txt

-   GitBook publishes /llms-full.txt with the full documentation content in one file, including hidden pages. \[primary source\] [source](https://gitbook.com/docs/publishing-documentation/llm-ready-docs), read 4 November 2026.
-   Scalar llms-full.txt concatenates the complete markdown of pages in the same navigation order as llms.txt. \[primary source\] [source](https://scalar.com/products/docs/configuration/llms-txt), read 4 November 2026.
-   Shopify serves /llms-full.txt as an alternate URL mirroring /agents.md. \[primary source\] [source](https://shopify.dev/docs/storefronts/themes/architecture/templates/agents-md-liquid), read 4 November 2026.
-   Mintlify exempts llms-full.txt from the 100,000-character split it applies to llms.txt. \[primary source\] [source](https://www.mintlify.com/docs/ai/llmstxt), read 4 November 2026.
-   docusaurus-plugin-llms writes llms-full.txt by bundling all docs content into one document at build time. \[primary source\] [source](https://github.com/rachfop/docusaurus-plugin-llms), read 4 November 2026.

## History and versions

-   v1 was published by Jeremy Howard on 2024-09-03. \[primary source, 2024-09-03\] [source](https://llmstxt.org/), read 4 November 2026.
-   v2 is dated August 2026; the llmstxt.org page shows a modified date of 2026-08-10. \[primary source, 2026-08-10\] [source](https://llmstxt.org/changes.html), read 4 November 2026.
-   v2 removed the llms\_txt2ctx tool from the proposal and says agents view or search llms.txt and then follow relevant links. \[primary source\] [source](https://llmstxt.org/changes.html), read 4 November 2026.
-   v2 defines subpath llms.txt files as covering pages under their path, with the most specific file applying. \[primary source\] [source](https://llmstxt.org/changes.html), read 4 November 2026.
-   Yoast added llms.txt generation to its free WordPress plugin in version 25.3, announced 2025-06-02. \[vendor claim, 2025-06-02\] [source](https://yoast.com/press/yoast-launches-the-first-llms-txt-generator-in-an-seo-plugin-ushering-in-a-new-era-of-ai-ready-content/), read 4 November 2026.
-   Chrome Lighthouse added an llms.txt audit to its Agentic Browsing category; a 404 is marked not applicable. \[primary source\] [source](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt), read 4 November 2026.

## The evidence on whether it matters

Ora ran one docs site in 19 configurations and probed it with a plain-fetch client and a JavaScript-running client on five topics (190 probes). Only two changes stopped retrieval, and neither was about llms.txt. The rest of the evidence comes from traced agent runs and from one site operator's logs.

-   Ora's lab found (2026-10-19) that across 19 configurations of one docs site and 190 probes, llms.txt, JSON-LD, sitemaps and markdown mirrors made no difference to whether an answer already in the HTML was retrieved. \[vendor claim, 2026-10-19\] [source](https://ora.ai/blog/agentready-launch?utm_source=agentreadyhtml.com&utm_medium=referral&utm_campaign=reference), read 4 November 2026.
-   Ora's lab found (2026-10-19) that 86 percent of llms.txt fetches arrived through a link, 93 percent for .well-known and 97 percent for openapi.json. \[vendor claim, 2026-10-19\] [source](https://ora.ai/blog/agentready-launch?utm_source=agentreadyhtml.com&utm_medium=referral&utm_campaign=reference), read 4 November 2026.
-   In the Vercel and Ora data, agents requested markdown on 2,259 of 3,470 web fetches (65.1 percent), and chose markdown in 95.7 percent of fetches that named any format. \[primary source, 2026-08-28\] [source](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents), read 4 November 2026.
-   Of 329 runs that reached llms.txt, 120 (36 percent) grounded the final answer in its content. \[primary source\] [source](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents), read 4 November 2026.
-   Cloudflare Radar found markdown content negotiation passing on 3.9 percent of scanned top domains at the April 2026 launch of its agent readiness score. \[primary source, 2026-04-17\] [source](https://blog.cloudflare.com/agent-readiness/), read 4 November 2026.
-   By default Cloudflare isitagentready.com checks markdown negotiation and leaves the llms.txt check opt-in. \[primary source\] [source](https://blog.cloudflare.com/agent-readiness/), read 4 November 2026.
-   A site operator saw ChatGPT, Claude on the web and Perplexity fetch HTML over 13,000 times and markdown once while answering people (4 to 18 September 2026). \[secondary source, 2026-09-22\] [source](https://www.goodbarber.com/blog/do-ai-agents-read-markdown-what-two-weeks-of-our-own-server-logs-say-a1622/), read 4 November 2026.

## Who generates it automatically

-   Docs platforms that generate llms.txt by default: Mintlify, GitBook, ReadMe, Fern, Redocly and Scalar Docs. \[primary source\] [source](https://scalar.com/products/docs/configuration/llms-txt), read 4 November 2026.
-   Wix auto-generates a default llms.txt that points agents to the Wix MCP endpoint. \[primary source\] [source](https://dev.wix.com/docs/api-reference/business-management/marketing/seo/txt-file-server/llms-txt/introduction), read 4 November 2026.
-   Shopify manages /llms.txt for every store as a mirror of /agents.md. \[primary source\] [source](https://shopify.dev/docs/storefronts/themes/architecture/templates/llms-txt-liquid), read 4 November 2026.
-   WordPress gets it through plugins: Yoast SEO, All in One SEO, Website LLMs.txt. \[primary source\] [source](https://developer.yoast.com/features/llms-txt/functional-specification/), read 4 November 2026.
-   Webflow and Framer serve an owner-supplied file but do not write it. \[primary source\] [source](https://help.webflow.com/hc/en-us/articles/43240104183315-Upload-an-llms-txt-file-to-your-site), read 4 November 2026.
-   Firecrawl generates a one-off file from a crawl; its hosted endpoint has been unmaintained since 2025-06-30. \[primary source\] [source](https://docs.firecrawl.dev/features/alpha/llmstxt), read 4 November 2026.

## Arguments against

-   Google Search says it does not use the file and that no special files are needed for AI Overviews or AI Mode. \[primary source\] [source](https://developers.google.com/search/docs/appearance/ai-features), read 4 November 2026.
-   Scrunch does not use llms.txt in AXP and says consumer AI systems do not reliably read it. \[vendor claim\] [source](https://helpcenter.scrunchai.com/en/articles/12845899-how-axp-serves-content-to-ai-agents), read 4 November 2026.
-   The Vercel and Ora guide says llms.txt, markdown mirrors and JSON-LD do not replace accurate, fetchable source pages, and agents can use llms.txt only after reaching the site. \[primary source\] [source](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents), read 4 November 2026.
-   Large single files exceed context windows; Cloudflare saw agents fall into a grep loop on oversized llms.txt files. \[primary source\] [source](https://blog.cloudflare.com/agent-readiness/), read 4 November 2026.
-   ReadMe calls llms.txt control over what an agent finds when it looks, not a guarantee that it will look. \[primary source\] [source](https://docs.readme.com/main/docs/llmstxt), read 4 November 2026.

## The numbers in one place

| Figure | Value | Who measured it |
| --- | --- | --- |
| Sites serving a plain-text llms.txt | 27.4% of 8,816 home pages that answered 200 | This site's crawl, 2026-11-04 |
| llms.txt files linked from the home page | 9.8% of 2,418 files | This site's crawl, 2026-11-04 |
| llms.txt fetches that arrived through a link | 86 percent (356 of 416 attributed fetches) | Ora's lab and the Vercel guide, 2026-10-19 |
| Web fetches that asked for markdown | 65.1 percent (2,259 of 3,470) | Vercel and Ora data, 2026-08-28 |
| Runs that reached llms.txt and then fetched a page it listed | 36 percent (117 of 325) | Vercel and Ora data |
| Markdown negotiation passing on scanned top sites | 3.9 percent | Cloudflare Radar, April 2026 |

## Find a file and check yours

Three data pages extend the table above: [what agents reach](https://agentreadyhtml.com/what-agents-reach/) sets Ora's two studies side by side, [llms.txt size](https://agentreadyhtml.com/llms-txt-size/) measures the files and [unlinked llms.txt](https://agentreadyhtml.com/unlinked-llms-txt/) lists sites that serve one and do not link it. [llms.txt examples that score well on Ora](https://agentreadyhtml.com/llms-txt-examples/) joins the index to Ora's cached site scores.

The [llms.txt Index](https://agentreadyhtml.com/llms-txt-index/) renders 401 files with their parse and grade. The [generator](https://agentreadyhtml.com/generator/) writes a file from a list of URLs. The [guide to writing one that gets followed](https://agentreadyhtml.com/guides/write-an-llms-txt-that-gets-followed/) has the structure and the linking. To rate a whole site against the same read-layer behaviours, the check we recommend is [Ora's scan](https://ora.ai/api/checks?utm_source=agentreadyhtml.com&utm_medium=referral&utm_campaign=llms-txt), whose Access layer lists 42 checks (Ora's catalogue, read 5 November 2026). Cloudflare's scanner at [isitagentready.com](https://isitagentready.com/) is a free quick alternative, and [Mintlify's score](https://www.mintlify.com/score) grades documentation.

## Sources

1.  [llms.txt proposal (llmstxt.org)](https://llmstxt.org/) https://llmstxt.org/, read 4 November 2026
2.  [llmstxt.org llms.txt](https://llmstxt.org/llms.txt) https://llmstxt.org/llms.txt, read 4 November 2026
3.  [Fern docs: llms.txt](https://buildwithfern.com/learn/docs/ai-features/llms-txt) https://buildwithfern.com/learn/docs/ai-features/llms-txt, read 4 November 2026
4.  [ReadMe docs: LLMs.txt](https://docs.readme.com/main/docs/llmstxt) https://docs.readme.com/main/docs/llmstxt, read 4 November 2026
5.  [Google Search Central documentation updates](https://developers.google.com/search/updates) https://developers.google.com/search/updates, read 4 November 2026
6.  [GitBook docs: LLM-ready docs](https://gitbook.com/docs/publishing-documentation/llm-ready-docs) https://gitbook.com/docs/publishing-documentation/llm-ready-docs, read 4 November 2026
7.  [Scalar docs: llms.txt](https://scalar.com/products/docs/configuration/llms-txt) https://scalar.com/products/docs/configuration/llms-txt, read 4 November 2026
8.  [Shopify: agents.md.liquid](https://shopify.dev/docs/storefronts/themes/architecture/templates/agents-md-liquid) https://shopify.dev/docs/storefronts/themes/architecture/templates/agents-md-liquid, read 4 November 2026
9.  [Mintlify docs: llms.txt](https://www.mintlify.com/docs/ai/llmstxt) https://www.mintlify.com/docs/ai/llmstxt, read 4 November 2026
10.  [docusaurus-plugin-llms](https://github.com/rachfop/docusaurus-plugin-llms) https://github.com/rachfop/docusaurus-plugin-llms, read 4 November 2026
11.  [llms.txt v1 to v2 changes](https://llmstxt.org/changes.html) https://llmstxt.org/changes.html, read 4 November 2026
12.  [Yoast press release: llms.txt generator](https://yoast.com/press/yoast-launches-the-first-llms-txt-generator-in-an-seo-plugin-ushering-in-a-new-era-of-ai-ready-content/) https://yoast.com/press/yoast-launches-the-first-llms-txt-generator-in-an-seo-plugin-ushering-in-a-new-era-of-ai-ready-content/, read 4 November 2026
13.  [Chrome Lighthouse: llms.txt audit](https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt) https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt, read 4 November 2026
14.  [Ora: AgentReady launch post, findings 01 to 04](https://ora.ai/blog/agentready-launch?utm_source=agentreadyhtml.com&utm_medium=referral&utm_campaign=reference) https://ora.ai/blog/agentready-launch, read 4 November 2026
15.  [Vercel and Ora: Make your site readable by AI agents](https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents) https://vercel.com/kb/guide/make-your-site-readable-by-ai-agents, read 4 November 2026
16.  [Cloudflare: agent readiness score and Radar numbers](https://blog.cloudflare.com/agent-readiness/) https://blog.cloudflare.com/agent-readiness/, read 4 November 2026
17.  [GoodBarber: Do AI agents read Markdown](https://www.goodbarber.com/blog/do-ai-agents-read-markdown-what-two-weeks-of-our-own-server-logs-say-a1622/) https://www.goodbarber.com/blog/do-ai-agents-read-markdown-what-two-weeks-of-our-own-server-logs-say-a1622/, read 4 November 2026
18.  [Wix Llms Txt API](https://dev.wix.com/docs/api-reference/business-management/marketing/seo/txt-file-server/llms-txt/introduction) https://dev.wix.com/docs/api-reference/business-management/marketing/seo/txt-file-server/llms-txt/introduction, read 4 November 2026
19.  [Shopify: llms.txt.liquid](https://shopify.dev/docs/storefronts/themes/architecture/templates/llms-txt-liquid) https://shopify.dev/docs/storefronts/themes/architecture/templates/llms-txt-liquid, read 4 November 2026
20.  [Yoast llms.txt functional specification](https://developer.yoast.com/features/llms-txt/functional-specification/) https://developer.yoast.com/features/llms-txt/functional-specification/, read 4 November 2026
21.  [Webflow help: Upload an llms.txt file](https://help.webflow.com/hc/en-us/articles/43240104183315-Upload-an-llms-txt-file-to-your-site) https://help.webflow.com/hc/en-us/articles/43240104183315-Upload-an-llms-txt-file-to-your-site, read 4 November 2026
22.  [Firecrawl docs: LLMs.txt generator endpoint](https://docs.firecrawl.dev/features/alpha/llmstxt) https://docs.firecrawl.dev/features/alpha/llmstxt, read 4 November 2026
23.  [Google Search Central: AI features](https://developers.google.com/search/docs/appearance/ai-features) https://developers.google.com/search/docs/appearance/ai-features, read 4 November 2026
24.  [Scrunch help: How AXP serves content to AI agents](https://helpcenter.scrunchai.com/en/articles/12845899-how-axp-serves-content-to-ai-agents) https://helpcenter.scrunchai.com/en/articles/12845899-how-axp-serves-content-to-ai-agents, read 4 November 2026

Last reviewed 4 November 2026

Edition

Crawl

2026-11-04

Page updated

2026-11-04

Method

How the viewer reads a page, how a file is graded and how a platform is scored: [the method page](https://agentreadyhtml.com/method/).

**Status codes**  
The file should answer `200` with a text type. A `404` is not a failure on its own; an HTML page at /llms.txt is.