---
title: "The crawl dataset: what home pages serve agents: Agent Ready HTML"
description: "8,816 home pages that answered 200, read 2026-11-04: H1, llms.txt, markdown, soft 404s, JSON-LD and OpenAPI, by category and platform."
canonical: https://agentreadyhtml.com/data/
site: "Agent Ready HTML"
last_updated: 2026-11-04
date: 2026-11-04
section: "Data"
---

# The crawl dataset: what home pages serve agents: Agent Ready HTML

> 8,816 home pages that answered 200, read 2026-11-04: H1, llms.txt, markdown, soft 404s, JSON-LD and OpenAPI, by category and platform.

Source: https://agentreadyhtml.com/data/
Site: Agent Ready HTML
Last reviewed: 2026-11-04

# The crawl dataset

In short

The crawl read 13,071 domains on 4 November 2026. Of 8,816 home pages that answered 200, 73.7 percent carry an H1 in the initial HTML, 27.4 percent serve an llms.txt, 9.1 percent answer markdown negotiation and 16.9 percent return 200 for a missing path.

Crawl edition 2026-11-04, page updated 2026-11-04. [How the viewer, the index and the scores work](https://agentreadyhtml.com/method/).

The crawl read 13,071 domains on 4 November 2026. 8,816 home pages answered 200, and 259 were skipped because robots.txt disallows the crawler.

## Headline measures

Percent of home pages that answered 200 (or of the stated base), 2026-11-04.

Text equivalent of this chart

| Group | Value (%) | Counts |
| --- | --- | --- |
| H1 in initial HTML | 73.7 | 6,500 of 8,816 |
| JSON-LD on home | 53.8 |  |
| Soft 404 on made-up path | 16.9 |  |
| llms.txt present | 27.4 |  |
| Markdown negotiation | 9.1 |  |
| llms.txt linked (of files) | 9.8 |  |

| Measure | Count | Of | Percent |
| --- | --- | --- | --- |
| home pages with an H1 in the initial HTML | 6,500 | 8,816 | 73.7 |
| home pages with at least 200 words of text in the initial HTML | 7,306 | 8,816 | 82.9 |
| home pages that are JavaScript shells | 307 | 8,816 | 3.5 |
| home pages whose domain serves a plain-text llms.txt | 2,418 | 8,816 | 27.4 |
| llms.txt files that start with an H1 | 2,391 | 2,418 | 98.9 |
| llms.txt files linked from the home page HTML | 237 | 2,418 | 9.8 |
| home pages whose domain serves a plain-text llms-full.txt | 605 | 8,816 | 6.9 |
| home pages that answer Accept: text/markdown with markdown | 798 | 8,802 | 9.1 |
| made-up paths that return 404 | 7,061 | 8,743 | 80.8 |
| made-up paths that return 200 (soft 404) | 1,475 | 8,743 | 16.9 |
| home pages that carry JSON-LD | 4,743 | 8,816 | 53.8 |
| home pages that link an OpenAPI file | 29 | 8,816 | 0.3 |
| home pages with a link rel alternate of type text/markdown | 220 | 8,816 | 2.5 |

Definitions and the crawl rules are on the [method page](https://agentreadyhtml.com/method/). A home page that answered 200 is one that returned 200 to a plain-fetch client that honours robots.txt.

## By category

Category is the Ora public leaderboard category where the domain is on it; the rest of the top 10,000 has none. Only categories with at least 8 home pages that answered 200 are shown. Columns are percent of the category's home pages that answered 200 with an H1 in the initial HTML, serving llms.txt, answering markdown negotiation, and returning 200 for a made-up path.

| Category | Home pages | H1 % | llms.txt % | Markdown % | Soft 404 % |
| --- | --- | --- | --- | --- | --- |
| Payments & Fintech | 389 | 88.2 | 49.6 | 18.6 | 11.1 |
| Developer Tools | 367 | 92.6 | 58.9 | 25.9 | 13.7 |
| Healthcare | 240 | 83.3 | 33.8 | 15.4 | 14.2 |
| Infrastructure | 226 | 94.7 | 58.8 | 17.7 | 10.7 |
| Marketing | 213 | 94.4 | 55.4 | 19.2 | 6.2 |
| AI & ML | 193 | 85.5 | 49.2 | 20.2 | 16.7 |
| Data & Analytics | 144 | 95.8 | 68.1 | 21.5 | 6.9 |
| Productivity | 142 | 89.4 | 54.2 | 17.0 | 13.5 |
| E-commerce | 141 | 86.5 | 44 | 15.6 | 15.1 |
| Communication & Support | 140 | 95 | 54.3 | 26.4 | 10.9 |
| Agent Tools | 129 | 93 | 66.7 | 34.9 | 7.8 |
| Hardware & Robotics | 101 | 88.1 | 36.6 | 11.9 | 20.0 |
| Security | 89 | 92.1 | 56.2 | 13.5 | 10.1 |
| Legal & Compliance | 88 | 94.3 | 52.3 | 18.2 | 7.0 |
| Education | 85 | 81.2 | 48.2 | 12.9 | 18.8 |
| Logistics & Supply Chain | 82 | 91.5 | 43.9 | 8.5 | 12.2 |
| Real Estate & Construction | 81 | 97.5 | 39.5 | 6.2 | 11.1 |
| Consumer | 71 | 77.5 | 29.6 | 21.1 | 20.0 |
| Identity & Auth | 68 | 80.9 | 55.9 | 14.7 | 10.3 |
| Gaming & Media | 57 | 64.9 | 43.9 | 12.3 | 26.8 |
| HR & Recruiting | 53 | 92.5 | 62.3 | 11.3 | 11.3 |
| Climate & Energy | 41 | 85.4 | 34.1 | 19.5 | 17.1 |
| CRM | 39 | 94.9 | 64.1 | 38.5 | 7.9 |
| Commerce Tools | 26 | 96.2 | 61.5 | 23.1 | 3.8 |
| News & Intelligence | 25 | 56 | 24 | 4.0 | 4.2 |
| Travel & Hospitality | 13 | 61.5 | 15.4 | 15.4 | 15.4 |
| Top 10,000, no category | 5,573 | 64.7 | 13.7 | 3.3 | 19.5 |

## By platform

Platform is read from the generator meta tag; 1,332 home pages that answered 200 declare one. Platforms with at least 10 such pages are shown.

| Platform | Home pages | H1 % | llms.txt % | Markdown % | Soft 404 % |
| --- | --- | --- | --- | --- | --- |
| WordPress | 411 | 80.8 | 26 | 5.6 | 14.5 |
| Framer | 191 | 88 | 41.9 | 93.7 | 5.2 |
| Drupal | 146 | 81.5 | 8.2 | 4.1 | 4.8 |
| Astro | 110 | 98.2 | 68.2 | 24.5 | 7.3 |
| HubSpot | 68 | 94.1 | 25 | 1.5 | 14.7 |
| Gatsby | 58 | 81 | 29.3 | 5.2 | 12.1 |
| Elementor | 30 | 83.3 | 36.7 | 3.3 | 6.7 |
| Wix | 29 | 72.4 | 93.1 | 0.0 | 3.4 |
| All | 27 | 85.2 | 48.1 | 3.7 | 7.4 |
| Hugo | 27 | 59.3 | 7.4 | 3.7 | 0.0 |
| TYPO3 | 19 | 89.5 | 10.5 | 0.0 | 0.0 |
| Craft CMS | 10 | 90 | 20 | 20.0 | 10.0 |
| Docusaurus | 10 | 90 | 40 | 0.0 | 10.0 |

## JSON-LD types on home pages

| Type | Home pages | Share |
| --- | --- | --- |
| Organization | 3,259 | 37.0% |
| WebSite | 3,191 | 36.2% |
| WebPage | 1,507 | 17.1% |
| BreadcrumbList | 871 | 9.9% |
| SoftwareApplication | 809 | 9.2% |
| FAQPage | 641 | 7.3% |
| ImageObject | 376 | 4.3% |
| NewsMediaOrganization | 218 | 2.5% |
| ItemList | 207 | 2.3% |
| CollectionPage | 200 | 2.3% |
| Corporation | 169 | 1.9% |
| VideoObject | 111 | 1.3% |
| Service | 107 | 1.2% |
| SiteNavigationElement | 104 | 1.2% |
| Product | 102 | 1.2% |

## llms.txt size

| Size | Files |
| --- | --- |
| Under 2 KB | 222 |
| 2 to 10 KB | 1,282 |
| 10 to 50 KB | 799 |
| 50 to 100 KB | 94 |
| 100 KB and over | 133 |

## More from the crawl

Each measure above has its own page: [soft 404s by platform and category](https://agentreadyhtml.com/soft-404s/), [markdown negotiation by platform](https://agentreadyhtml.com/markdown-negotiation-by-platform/), [JSON-LD types](https://agentreadyhtml.com/json-ld-types-on-home-pages/), [the cost of client-side rendering](https://agentreadyhtml.com/client-side-rendering-and-agents/), [llms.txt size](https://agentreadyhtml.com/llms-txt-size/) and [llms.txt files that no page links to](https://agentreadyhtml.com/unlinked-llms-txt/). [Ora's Access layer](https://agentreadyhtml.com/ora-access-layer/) lists the checks that read the same behaviours, with the pass rates Ora publishes.

## Download

-   [crawl-measures-2026-11-04.csv](https://agentreadyhtml.com/data/crawl-measures-2026-11-04.csv) and [crawl-measures-2026-11-04.json](https://agentreadyhtml.com/data/crawl-measures-2026-11-04.json): the headline measures, by category and by platform.
-   [llms-txt-index-2026-11-04.csv](https://agentreadyhtml.com/data/llms-txt-index-2026-11-04.csv) and [llms-txt-index-2026-11-04.json](https://agentreadyhtml.com/data/llms-txt-index-2026-11-04.json): one row per llms.txt file found.
-   [llms-txt-entries-2026-11-04.json](https://agentreadyhtml.com/data/llms-txt-entries-2026-11-04.json): the parsed entries behind the entry pages, with the six structure checks.
-   [platform-scores-2026-11-04.json](https://agentreadyhtml.com/data/platform-scores-2026-11-04.json): the rubric and the score, evidence, source and read date of each platform.

The same files are queried through the [API and MCP server](https://agentreadyhtml.com/developers/).

Cite this: Agent Ready HTML. The crawl dataset. Edition 2026-11-04. https://agentreadyhtml.com/data/

For comparison, Cloudflare Radar found markdown negotiation passing on 3.9 percent of scanned top domains at the April 2026 launch of its agent readiness score; this site's crawl measured 9.1 percent on 4 November 2026. The two use different samples and different clients, so the numbers agree in order of magnitude, not in digits.

## Sources

1.  [Tranco list 746WX (crawl universe)](https://tranco-list.eu/list/746WX) https://tranco-list.eu/list/746WX, read 4 November 2026
2.  [Cloudflare: agent readiness score and Radar numbers](https://blog.cloudflare.com/agent-readiness/) https://blog.cloudflare.com/agent-readiness/, read 4 November 2026

Last reviewed 4 November 2026