# llms.txt adoption across 10 AI and dev-infrastructure sites

A snapshot probe of llms.txt presence at the engine and platform operators most likely to ship it. Four of ten do. Two block our scanner. One returns its SPA shell. The convention is real but adoption is uneven.

By AgentSite · 5 min read · Updated 2026-05-23

On 2026-05-23 we probed ten major AI and developer-infrastructure sites for `/llms.txt` presence. Four ship a real file. Two block our scanner outright. Three don't ship one. One returns its single-page-app shell at the path. The convention is real but adoption is uneven even among the engine and platform operators who would most directly benefit.

## What we measured

For each site, a single `curl` request to `https://<host>/llms.txt`, following one redirect, with a non-AI-bot user-agent (`AgentSiteScanner/1.0`). We recorded the final HTTP status code and the response size in bytes; for 200 responses we also sampled the first few hundred bytes to verify the content was actually a markdown `llms.txt` rather than an HTML SPA shell at the same path.

The probe is small (N=10) and snapshot-style — one moment in time, one user-agent, one tool. It is not a longitudinal study. It is what we observed today.

## The data

| Site | Status (after redirect) | Real `llms.txt`? |
| --- | --- | --- |
| `vercel.com/llms.txt` | 200, 167 KB | Yes — curated docs index ([sample](https://vercel.com/llms.txt)) |
| `nextjs.org/llms.txt` | 200, 7.7 KB | Yes — spec-shape markdown ([sample](https://nextjs.org/llms.txt)) |
| `www.cloudflare.com/llms.txt` | 200, 16.9 KB | Yes ([sample](https://www.cloudflare.com/llms.txt)) |
| `platform.claude.com/llms.txt` | 200, 167 KB | Yes — Anthropic's developer docs ship one |
| `www.anthropic.com/llms.txt` | 404 | No (marketing surface) |
| `schema.org/llms.txt` | 404 | No |
| `en.wikipedia.org/llms.txt` | 404 | No |
| `claude.ai/llms.txt` | 200, 4.5 KB | **No — SPA shell returned at the path** |
| `openai.com/llms.txt` | 403 | Blocked at edge |
| `www.perplexity.ai/llms.txt` | 403 | Blocked at edge |

## What the four who ship it have in common

Each of the four sites that returned a real `llms.txt` ships it on the **documentation** surface, not the marketing root. Vercel publishes both `/llms.txt` and `/docs/llms-full.txt`, mirroring the spec's recommended split. Next.js does the same with `/llms.txt` pointing at `/docs/llms.txt` and `/docs/llms-full.txt`. Anthropic publishes one on the developer platform host (`platform.claude.com`) while the marketing-host root (`anthropic.com`) returns 404.

The pattern: ship `llms.txt` where the documentation lives. The marketing root doesn't need it; the documentation surface absolutely does. Anyone fetching `<docs-host>/llms.txt` is plausibly an agent trying to answer a developer's question. Anyone fetching `<marketing-host>/llms.txt` is plausibly a probe like ours.

## The two 403s

Two sites returned 403 to our scanner user-agent: `openai.com` and `www.perplexity.ai`. Both are commercial AI engine operators with active anti-abuse postures at the CDN edge. The block is almost certainly a bot-policy default (not a deliberate llms.txt-specific rule) — the same UA pattern would return 403 on most paths on those domains. See [SSR-junk and bot walls](/ssr-junk-bot-wall) for the broader pattern.

The point worth surfacing: we cannot determine from outside whether these sites ship an `llms.txt`. They might. They might not. The probe is mute on it. For an AEO purpose, the answer doesn't matter — what matters is that the file isn't reachable to a scanner that doesn't carry one of the allowlisted user-agents.

## The SPA-shell catch

`claude.ai/llms.txt` returned status 200 with a 4.5 KB body. The body is HTML — the same React app shell `claude.ai` serves on every catch-all route. There is no `llms.txt` content at the path; the SPA shell is filling in. An agent fetching this URL receives a valid HTTP response but extracts zero useful content.

This is a Layer-1 failure mode on the consumer surface of the engine whose user-initiated-fetch bot (`Claude-User`) is one of the populations AEO sites optimize for. The product company building the engine that reads other people's sites returns an SPA shell at `/llms.txt` on its own consumer URL.

## What this tells us

Three observations:

1.  **Adoption is real but uneven.** Forty percent of our sample ships a working `llms.txt`. That is more than zero and less than universal. The convention is in the early-adopter phase among the operators with the strongest reason to ship it.
2.  **Docs surfaces lead; marketing surfaces lag.** Of the four real files, all live on documentation hosts. The pattern fits the spec's intent — `llms.txt` is a documentation-index file, not a marketing artifact.
3.  **The convention works.** Of the four real files we read, none was malformed. The spec is simple enough that operators who ship it ship it correctly.

For an AEO-targeted site in May 2026, `llms.txt` is still a differentiator rather than table stakes. The canonical definition of what the file is and how it's structured sits in [llms-txt](/llms-txt); the broader thesis on why the file matters at all sits in [agent readability](/agent-readability).

## Limits of this measurement

The probe is honestly small. Ten sites is not a population; one user-agent is not a fleet; one tool run is not a longitudinal observation. A larger probe with multiple user-agents at multiple times of day across multiple geographies would catch behavior our snapshot misses — particularly for the 403-returning sites, where a different UA could land 200. The Vercel "Rise of the AI Crawler" analysis is a better template for that kind of scale work ([Vercel, Dec 2024](https://vercel.com/blog/the-rise-of-the-ai-crawler)); ours is a directional read, not a measurement.

We will re-run this probe in three months and append the deltas to this page.