---
title: "API and MCP server: Agent Ready HTML"
description: "How to query the llms.txt Index, the crawl measures and the platform scores: the MCP server at /mcp, the REST API in openapi.json and the NLWeb ask endpoint."
canonical: https://agentreadyhtml.com/developers/
site: "Agent Ready HTML"
last_updated: 2026-11-04
section: "Developers"
---

# API and MCP server: Agent Ready HTML

> How to query the llms.txt Index, the crawl measures and the platform scores: the MCP server at /mcp, the REST API in openapi.json and the NLWeb ask endpoint.

Source: https://agentreadyhtml.com/developers/
Site: Agent Ready HTML
Last reviewed: 2026-11-04

# API and MCP server

In short

The llms.txt Index, the crawl measures and the platform scores are also data an agent can query: an MCP server at /mcp with five read-only tools, a REST API under /api/v1 described by openapi.json, and an NLWeb /ask endpoint. Every answer names the page it comes from, and nothing needs a credential.

## Connect the MCP server

The server speaks the Streamable HTTP transport at `https://agentreadyhtml.com/mcp`. Send JSON-RPC 2.0 messages with POST and read the JSON answer. There is no session to open and no credential to send, so one request gets one answer. An MCP client that takes a URL connects with that address alone; the [server card](https://agentreadyhtml.com/mcp/server-card) and the [manifest](https://agentreadyhtml.com/.well-known/mcp.json) say the same thing in the forms clients read.

```bash
curl -s https://agentreadyhtml.com/mcp -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"my-agent","version":"1.0"}}}'
```

```bash
curl -s https://agentreadyhtml.com/mcp -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream' -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"get_llms_entry","arguments":{"domain":"supabase.com"}}}'
```

A tool answers with the result as `structuredContent` and the same JSON as text. A wrong argument answers `isError: true` with the parameter list, and an unknown tool answers JSON-RPC error -32602. The server also lists five resources with `resources/list`: the page index, the methodology, this page and two data files, readable with `resources/read`.

## The five tools

| Tool | What it returns | Parameters |
| --- | --- | --- |
| search_llms_index | Lists llms.txt files from the index (domain, Tranco rank, category, platform, grade A to F, size, H1, link counts, whether the home page links it, page URL). Filter by words in the domain or H1, grade, category, platform, minimum links or linked_from_home. Call it to find which sites serve an llms.txt and how it grades; use get_llms_entry for one domain in full. | q?: Words to look for in the domain or the H1 of the file; every word must appear, for example "stripe" or "developer docs".; grade?: Structure grade: A passes all six checks of the llmstxt.org structure, F two or fewer.; category?: Category of the domain as the index lists it, for example "Developer Tools". grade_distribution with group_by category lists them all.; platform?: Platform the home page declares in its generator tag, for example WordPress, Framer or Astro.; min_links?: Keep files that list at least this many links.; linked_from_home?: true keeps files the home page links; false keeps files it does not link. Files whose home page was not read are left out of both. |
| get_llms_entry | Returns one domain from the index: its row (grade, checks passed, size, H1, links, whether the home page links the file), what the grade means, and, when the domain has an entry page, the parsed summary, H2 sections, link hosts and the six structure checks with their detail. Call it after search_llms_index, or with any domain you are checking. | domain: A domain such as stripe.com, or a URL on it; a leading www. is ignored if the domain is not listed with it. |
| grade_distribution | Returns how many llms.txt files earned each grade (A to F), overall and grouped by category or by platform, with the share that earned an A. Call it for adoption and quality questions across the index, or to list the category and platform names that search_llms_index accepts. | group_by?: category (default), platform, or none for the overall counts only.; min_files?: Leave out groups with fewer files than this; platform names read from generator tags are many and small, so 5 is a useful floor. |
| get_crawl_measure | Lists the crawl measures (count, denominator and percent) for home pages that answered 200: headline figures, by category and by platform, for example the share that serve llms.txt, answer Accept: text/markdown with markdown or return 200 for a made-up path. Filter by group_type, group and words of the measure. Call it for how common a practice is. | group_type?: headline is the whole crawl; category and platform split it.; group?: The group name, for example "Developer Tools" (a category) or "WordPress" (a platform); case-insensitive. Omit group_type to match the name in any type.; measure?: Words from the measure name; every word must appear, for example "llms.txt" or "markdown" or "soft 404". |
| list_platforms | Lists the 23 platforms (docs platforms, site builders, frameworks, edge products) with their total on the nine-key rubric and each key score (2 on by default, 1 with setup, 0 not offered, n/a). With id, returns one platform with the evidence, source URL and read date for every key; with key, adds that key's evidence to each platform. Call it to compare platforms or to cite one score. | id?: Platform id for one platform in full, for example astro, nextjs-app-router or mintlify.; type?: Keep one kind of platform.; key?: A rubric key to rank by and to show the evidence for, for example markdown_negotiation. |

`search_llms_index`, `get_crawl_measure`, `list_platforms` are paginated: they take `limit` (1 to 100, default 20) and the `next_cursor` of the previous page as `cursor`, and answer with `items`, `total` and `next_cursor`.

## The REST API

Each tool is also one GET endpoint under `/api/v1/`, with the parameters in the query string and the same JSON in the answer. [/api](https://agentreadyhtml.com/api) is the index. [/openapi.json](https://agentreadyhtml.com/openapi.json) describes every operation with its parameters, response schema and the Error schema, and [/.well-known/api-catalog](https://agentreadyhtml.com/.well-known/api-catalog) links that description the way RFC 9727 asks. The examples below return real results.

-   [/api/v1/search\_llms\_index?category=Developer+Tools&grade=A](https://agentreadyhtml.com/api/v1/search_llms_index?category=Developer+Tools&grade=A): search the llms.txt Index, paginated.
-   [/api/v1/get\_llms\_entry?domain=supabase.com](https://agentreadyhtml.com/api/v1/get_llms_entry?domain=supabase.com): one domain in the llms.txt Index.
-   [/api/v1/grade\_distribution?group\_by=category&min\_files=20](https://agentreadyhtml.com/api/v1/grade_distribution?group_by=category&min_files=20): grades across the llms.txt Index.
-   [/api/v1/get\_crawl\_measure?group\_type=headline&measure=llms.txt](https://agentreadyhtml.com/api/v1/get_crawl_measure?group_type=headline&measure=llms.txt): crawl measures of the read layer, paginated.
-   [/api/v1/list\_platforms?type=framework](https://agentreadyhtml.com/api/v1/list_platforms?type=framework): platforms scored on the read layer, paginated.
-   [/api](https://agentreadyhtml.com/api): the index of endpoints, data files, limits and versioning.

```bash
curl -s 'https://agentreadyhtml.com/api/v1/get_llms_entry?domain=supabase.com' | jq '{grade, links, linked_from_home}'
```

## Ask in plain words

The `/ask` endpoint follows NLWeb 0.55. It searches the llms.txt entries and rows, the crawl measures, the platform scores and every page listed in [/llms.txt](https://agentreadyhtml.com/llms.txt), and returns schema.org objects, each with a `grounding.source` URL to cite. GET takes `query`; POST takes the NLWeb request object. Add `streaming=true`, or `prefer.streaming` in the body, for server-sent events named start, result and complete. `mode=summarize` adds one sentence assembled from the results, never written by a model.

```bash
curl -s 'https://agentreadyhtml.com/ask?query=markdown+negotiation+on+Next.js'
```

```bash
curl -s https://agentreadyhtml.com/ask -H 'Content-Type: application/json' -d '{"query":{"text":"markdown negotiation on Next.js"},"prefer":{"mode":"list"},"meta":{"version":"0.55"}}'
```

A query that matches nothing answers `_meta.response_type: failure` with code NO\_RESULTS, and a missing query answers 400 with INVALID\_QUERY. A context object is accepted and ignored, because the answers come from the data and not from a conversation.

## Data files behind the tools

The tools read these published files at request time, so a tool answer and the page it cites come from the same edition. A reader who wants everything can download the files whole instead of paging the API.

-   [/data/llms-txt-index-2026-11-04.json](https://agentreadyhtml.com/data/llms-txt-index-2026-11-04.json): one row per llms.txt file found in the crawl, also as [CSV](https://agentreadyhtml.com/data/llms-txt-index-2026-11-04.csv).
-   [/data/llms-txt-entries-2026-11-04.json](https://agentreadyhtml.com/data/llms-txt-entries-2026-11-04.json): the parsed entries behind the entry pages, with the six structure checks.
-   [/data/crawl-measures-2026-11-04.json](https://agentreadyhtml.com/data/crawl-measures-2026-11-04.json): the crawl measures with counts and denominators, also as [CSV](https://agentreadyhtml.com/data/crawl-measures-2026-11-04.csv).
-   [/data/platform-scores-2026-11-04.json](https://agentreadyhtml.com/data/platform-scores-2026-11-04.json): the rubric and the scores, evidence, sources and read dates of the platforms.

## Discovery files

-   [/openapi.json](https://agentreadyhtml.com/openapi.json): OpenAPI 3.1 for the REST API and /ask, linked from every page as rel=service-desc and from the Link header.
-   [/.well-known/api-catalog](https://agentreadyhtml.com/.well-known/api-catalog): an RFC 9727 linkset that points to the descriptions of the API, the MCP server and the ask endpoint.
-   [/mcp/server-card](https://agentreadyhtml.com/mcp/server-card): the MCP server card with name, version, remote URL and protocol versions, with a copy at /.well-known/mcp/server-card.json.
-   [/.well-known/mcp.json](https://agentreadyhtml.com/.well-known/mcp.json): the manifest with the endpoint, the transport and the tools with their input schemas.
-   [/.well-known/ai-catalog.json](https://agentreadyhtml.com/.well-known/ai-catalog.json): AI Catalog 1.0 with entries for the MCP card, the OpenAPI document, the ask endpoint and every document; the same file is at /.well-known/ard.json.
-   [/schemamap.xml](https://agentreadyhtml.com/schemamap.xml): the schema feeds, which carry the entries, rows, platforms and measures as JSON-LD lines under /feeds/ and are named in robots.txt.

## Access, limits and errors

Credential

None. The OpenAPI document declares `security: []` with no security schemes, and [auth.md](https://agentreadyhtml.com/auth.md) says the same. The site publishes no OAuth metadata because no endpoint takes a token; metadata for an authorization server that does not exist would point agents at a door that is not there.

Rate limit

120 requests a minute per client address, counted per running instance and announced in `RateLimit` and `RateLimit-Policy` headers. A 429 carries `Retry-After`, and the host adds limits of its own above that. The whole index is one download from [/data/](https://agentreadyhtml.com/data/), so bulk readers never need the API.

Errors

JSON, never HTML: `{"error":{"code","message","hint","status","docs"}}` with the codes invalid\_params, not\_found, method\_not\_allowed, rate\_limited and internal. A missing endpoint under /api answers 404 and lists the endpoints in the hint.

Writes

There are none, so every call is safe to retry and no idempotency key exists. Every call answers in one response, so there is no job to poll.

CORS

Every answer carries `Access-Control-Allow-Origin: *`, so an agent that runs in a browser can call the API and the MCP server.

## Versioning

The path prefix `/api/v1` is the contract. A breaking change, such as a field removed or renamed or a tool removed, ships under `/api/v2` while `/api/v1` keeps answering for at least 180 days, announced on this page and in the `info.version` of openapi.json. Additive changes, such as a new field or a new tool, keep the prefix. The MCP server follows the same rule and carries the same version number in its card and in `serverInfo.version`. The data behind every answer is edition 2026-11-04; the edition is in the `X-Dataset-Edition` header and in every tool result.

## Cite the data

Every result names the page or file it comes from; cite that URL. The llms.txt Index and the crawl datasets are licensed [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) under the [terms](https://agentreadyhtml.com/terms/), and the other text and data may be quoted with attribution and a link to the page. The citation line for the data is on the [data page](https://agentreadyhtml.com/data/).

## What the surface does not do

It does not write, hold accounts, fetch other sites or generate text with a model. For a live look at one URL, use the [Plain Fetch viewer](https://agentreadyhtml.com/viewer/); the tools answer from the crawl of 2026-11-04, which can differ from what a site serves today. The pages stay the record, and the [methodology](https://agentreadyhtml.com/method/) explains how the rows are made.

## Sources

1.  [MCP specification, Streamable HTTP transport](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports) https://modelcontextprotocol.io/specification/2025-06-18/basic/transports, read 4 November 2026
2.  [NLWeb specification v0.55](https://github.com/nlweb-ai/website/blob/main/NLWEBSPEC.md) https://github.com/nlweb-ai/website/blob/main/NLWEBSPEC.md, read 4 November 2026
3.  [RFC 9727, api-catalog](https://www.rfc-editor.org/rfc/rfc9727) https://www.rfc-editor.org/rfc/rfc9727, read 4 November 2026
4.  [OpenAPI 3.1.0](https://spec.openapis.org/oas/v3.1.0) https://spec.openapis.org/oas/v3.1.0, read 4 November 2026
5.  [Schema Feeds specification 0.1](https://github.com/nlweb-ai/website/blob/main/SCHEMA_SPEC.md) https://github.com/nlweb-ai/website/blob/main/SCHEMA\_SPEC.md, read 4 November 2026

Last reviewed 4 November 2026

Edition

Crawl

2026-11-04

Page updated

2026-11-04

Method

How the viewer reads a page, how a file is graded and how a platform is scored: [the method page](https://agentreadyhtml.com/method/).