76% of Sites With an llms.txt File Get It Wrong

Of the 142 websites we audited between May 4 and July 19, 2026, 79 had no llms.txt file at all. Of the 63 that published one, 48 published a broken one. That leaves 13 sites — 9% of the sample — with a valid file.

The llms.txt proposal is simple: put a markdown file at your domain root that gives AI agents a curated, plain-text map of your site. When someone asks ChatGPT, Claude, or Perplexity a question your site could answer, an agent that fetches your llms.txt at inference time gets exactly the context you chose — instead of a best-effort scrape of your homepage HTML. The specification is deliberately minimal: an H1 with your site's name, a blockquote summary, and markdown lists of links to your most important content.

A year into adoption, the gap is no longer awareness. It is execution.

The dataset

Methodology: latest completed audit snapshot per domain, drawn from production checks between May 4 and July 19, 2026. All domains anonymized. The sample is self-selected — sites whose owners ran an audit — and skews toward small and mid-size properties, so treat it as directional rather than a census of the web.

The 3 mistakes behind nearly every invalid file

Our engine validates an llms.txt against the spec's structure, and the failure evidence is remarkably consistent. 3 issues account for almost everything — most broken files have 2 or 3 of them at once:

1. No reference links (40 of 48 invalid files). The file exists, but it contains no markdown links. This is the entire point of the format — a curated list of URLs an agent should read. A file without links is a poster with no map.

2. Missing the summary blockquote (38 of 48). The spec calls for a > Summary blockquote right after the title that tells an agent what the site is and how to interpret the links. Without it, an agent gets a list of URLs with no frame for choosing between them.

3. Missing the H1 title (33 of 48). The only section the spec marks as strictly required. Files that open with prose, a comment, or an H2 give parsers nothing to anchor on.

The pattern underneath all 3 is the same: teams generate a placeholder — sometimes literally the word "llms.txt" repeated, or a paragraph of marketing copy — and deploy it as a checkbox. An agent reading such a file learns nothing and moves on. The fetch was wasted.

When the links themselves are dead

Validity is structure. The next check is whether the referenced URLs actually resolve. Most sites with valid structure pass this too, but the failures are instructive: 7 sites listed links that returned errors, with a median of 19 dead references per site. One ecommerce property listed 3,042 product URLs in its llms.txt that no longer resolved — the file had clearly been generated once from a product feed and never regenerated.

That anecdote is a sample of 7, so treat it as a warning rather than a statistic. But the mechanism is worth remembering: an llms.txt is a living document. If your catalog, docs, or pricing pages move, the file rots — and an agent following dead links may conclude your content is gone, not moved.

What a valid file looks like

The spec's own example is the best template. A minimal, valid llms.txt needs an H1, a summary blockquote, and at least one list of annotated links:

# Acme Analytics

> Acme Analytics is a reporting API for ecommerce teams. This file lists our most useful resources for AI agents.

## Docs

- [API quickstart](https://acme.example/docs/quickstart): Authentication and first request in 5 minutes
- [Metrics reference](https://acme.example/docs/metrics): Every metric, its definition, and its SQL

## Product

- [Pricing](https://acme.example/pricing): Plans and usage limits
- [Changelog](https://acme.example/changelog): Release notes since 2024

## Optional

- [Full documentation index](https://acme.example/docs): Everything else, for larger context windows

The Optional section has a defined meaning: agents short on context may skip it. Put your best material above it.

Why this is worth 30 minutes

Sitemaps tell search engines what exists. robots.txt tells crawlers what is allowed. llms.txt is the only file that tells an AI agent what matters — written in the format it natively reads, at the moment it is deciding whether your site helps answer a user's question. It is also one of the few AI-search levers you fully control: no model provider, no ranking algorithm, no waiting for a crawl.

The fix sequence is short. Write the file from the template above — 10 minutes. Link only to pages that resolve, and prefer stable URLs. Regenerate or review it whenever your information architecture changes. Then verify it the way an agent would: fetch it, parse it, follow the links.

Every audit we run checks all 3 layers — presence, validity, and whether the references resolve — as part of the AI readiness family. If you want to know which of the 3 groups your site falls into, you can run your own report and find the llms.txt checks in the AI readiness section.

The bottom line

Adoption is the first gap: 56% of audited sites have no file. But the sharper finding is that publishing a file is not the finish line — 3 in 4 of the sites that bothered published one an agent cannot use. The sites in the valid minority did nothing exotic. They wrote an H1, a 2-sentence summary, and a list of links that work. That is the entire bar — and 91% of the sample is under it.