13% of Audited Sites Block an AI Crawler — Correcting Our Own 40.8%

Updated August 31, 2026. The version of this article published on August 20 led with "20 of the 49 sites we audited — 40.8%." That number was wrong twice over. 40.8% was the share of individual audit runs that failed the check, not the share of distinct sites, and multiplying it back onto a site count produced a "20 sites" that nobody had ever counted. Worse, when we went back and re-read every robots.txt in the sample against the specification, 2 defects in our own parser turned out to be manufacturing half of the failures. The corrected figures are below, and the correction is now the more useful half of this article.

Every audit we run fetches robots.txt and reads it the way an AI crawler would. On August 31, 2026 we re-fetched the file from all 48 distinct domains audited in our current engine window, and 45 of them served one. Of those 45, 6 sites — 13.3% — shut at least 1 major AI crawler out of the entire site. At the opposite extreme, 8 sites have no rules that apply to AI crawlers at all — not even a wildcard directive for them to obey. That is 17.8% of the sample.

That leaves 31 sites — 68.9% of the sample — with a robots.txt that states a policy an AI crawler will actually read and blocks none of them sitewide. In a year when ChatGPT search, Perplexity, and Claude cite sources by fetching them, the file most owners have not opened since launch has quietly become a visibility switch. Fewer people have flipped it off than we first reported. More of them have never touched it at all.

Methodology. The population is every distinct domain with a completed audit in our current engine window, which runs from May 23 to August 31, 2026. That is 48 domains across 228 report runs. The stored verdicts carry the defective parser's answer, so the table above is not the stored tally. It is a same-day re-fetch of those domains' live robots.txt files, read on August 31 and evaluated with the corrected parser. 3 of the 48 no longer serve a robots.txt at all, which is why the denominator is 45. The sample is self-selected — these are sites somebody chose to run an audit on — and it skews small-to-mid-size. Treat the rates as directional for the long tail of the web rather than the web at large. Domains are not named.

What our parser got wrong, and how we found it

The check itself is simple. Our engine parses every robots.txt group and evaluates 6 user agents against it: GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Claude-SearchBot, and CCBot. A crawler counts as blocked only when the group that governs it carries a sitewide Disallow: /. Partial disallows like Disallow: /admin register as a stated policy, which is healthy, and they pass.

Applying that definition correctly is where we failed. Re-reading all 45 files turned up 2 distinct defects, pulling in opposite directions.

An empty Disallow: is not a block. RFC 9309 is explicit that a rule with no path is ignored, which makes Disallow: the canonical way to say allow everything. Yoast emits exactly that file by default, and so do several shared hosts. Our parser treated the empty value as equivalent to / and flagged all 6 crawlers as blocked. 5 of the 10 sites the engine had flagged were running an ordinary allow-all file: 3 of them the Yoast default block verbatim, 1 of them a bare 2-line file.

A named group overrides the wildcard. RFC 9309 also says a crawler obeys the single most specific group that names it and ignores User-agent: * entirely when such a group exists. Our parser read every matching group in file order and let the last one win. That cuts both ways. A large social platform in the sample names GPTBot, ClaudeBot, and PerplexityBot with narrow path rules and then closes the file with a catch-all User-agent: * / Disallow: / — our parser handed those 3 named crawlers the catch-all they were exempt from. And 1 site was miscounted in the other direction: it disallows 3 AI crawlers by name near the top of its file, then states looser wildcard rules further down, and our parser let the wildcard erase a real, deliberate block. That site had been passing.

Both defects are now fixed, with unit tests built from the exact files that exposed them. The net effect on this sample: 5 sites the old parser called blocked are not, and 1 site it called clean is. The 3 numbers side by side, all from the same window: the engine's stored verdicts read 42.9% of 226 individual audit runs and 21.3% of 47 sites; re-reading the live files with the corrected parser gives 13.3% of 45 sites. Only the last one is a claim about sites, measured with a parser that reads the specification correctly.

One property of the check survives all of this: it is still conservative. A WAF challenge, a CDN bot rule, or a firewall block never appears in robots.txt, so 13.3% remains a floor on how many sites actually turn these crawlers away. And a site can still block a crawler without ever typing its name — a blanket wildcard disallow written for a preview environment applies to GPTBot exactly as it applies to everything else. 2 of the 6 confirmed blockers reach at least 1 crawler that way: a preview host whose entire robots.txt is a 3-line blanket disallow, and a large platform whose closing catch-all sweeps up the 3 crawlers it did not name.

A rule written in 2023 answers the wrong question in 2026

Our corrected rate now sits below what larger external samples report. A March 2026 analysis of 10,000 sites by SEO Score Tools found 18.7% actively block GPTBot and 41.3% carry no AI-specific robots rules at all. Both of their figures land above ours — definitions differ between samples, our sample is a fraction of the size, and our check counts a wildcard group as an applied policy where theirs looks for named ones. We are not claiming our 13.3% overturns their 18.7%; a 45-site sample cannot. What both datasets agree on is the shape: a minority of the web has made an AI-access decision, a further slice has made one without knowing it, and the rest have not been asked.

The unconsidered part is the story, and our corrected data is careful about which part that is. The 6 blocks we confirmed are mostly deliberate — 4 name the crawler, and 2 of those run Cloudflare's managed content-signals list, which is a policy written this year and not a leftover. What we cannot show from 45 sites is how much of the wider web is still carrying a stale rule; that is what the external samples above are for.

The mechanism is worth stating anyway, because it is what makes a stale rule expensive. In 2023, when GPTBot first appeared in server logs and CCBot became newly notorious as a training source, the only known consequence of access was model training — so bot-block lists circulated, plugins shipped 1-click toggles, and robots.txt files inherited disallows that nobody has re-read since. The question those rules answered was "do I want my content in a training corpus?" The question that matters in 2026 is different: "do I want to be findable and citable when an assistant answers on my topic?" A rule written for the first question now silently answers the second.

Training crawlers and search crawlers deserve different answers

The 6 crawlers we probe split cleanly into 2 jobs, per each operator's published bot documentation:

• Search and retrieval. OAI-SearchBot indexes content so ChatGPT search can surface and link to it — OpenAI documents that this access, and only this access, governs whether pages are considered for ChatGPT search results. PerplexityBot builds Perplexity's search index. Claude-SearchBot indexes to improve Claude's search-backed answers. Blocking any of these removes you from that assistant's citations.
• Training. GPTBot gathers content that may train OpenAI's models. ClaudeBot does the equivalent crawling for Anthropic. CCBot feeds Common Crawl, the open corpus behind many research and training datasets. OpenAI is explicit that blocking GPTBot has no effect on ChatGPT search inclusion — the 2 pipelines are separate.

Google runs the same split under different names: its AI surfaces ride on ordinary Googlebot access, while the separate Google-Extended token controls Gemini training and grounding. We covered that machinery in our guide to Google's generative search.

The split turns a vague anxiety into a 2-part decision. Blocking training crawlers is a legitimate stance on how your content may be reused, and it costs you nothing in AI search. Blocking retrieval crawlers is a visibility decision — the same category as noindexing yourself — and it deserves the same deliberateness.

Write a robots.txt that records the decision

Here is a policy that stays fully visible in AI search while withholding training consent — the most common deliberate split we see:

# Search & retrieval — open, so assistants can find and cite this site
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Training — withheld; this has no effect on AI search visibility
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

If you want everything open, say so explicitly rather than by omission — a named Allow: / group per crawler documents that someone decided, which is exactly what the 17.8% with no applicable rules are missing.

3 verification steps close the loop. Fetch https://yoursite.com/robots.txt from outside your network and read what production actually serves — CDNs and platforms can inject or override the file in your repository, and some edge providers ship 1-click AI-bot blocking that overrules your directives entirely. Check any WAF or bot-management rules for the same 6 user agents, because robots.txt permission means nothing to a crawler that gets a 403. Then re-run the check after every infrastructure change; a provider migration can flip this switch without touching your code.

1 note on how our audit reads the deliberate policy above: the sitewide disallows on GPTBot, ClaudeBot, and CCBot will still surface as a finding. That is by design. A sitewide AI-crawler block should always be an affirmed decision, and the finding is where you affirm it — the failure mode this check exists to catch is the block nobody remembers writing.

The cheapest visibility audit you will run this year

robots.txt is 1 of 2 files through which your site speaks to AI systems — the other is llms.txt, where our data shows 76% of files fail the agents they were written for. Both share the same failure signature as the blocks in this report: written once, syntactically plausible, never read back. So did our parser, which is the uncomfortable part of this correction — a rule that has not been re-read since the day it was written is exactly what this check exists to catch, and ours had not been re-read either.

The corrected picture is also a more encouraging one than the headline we first ran. 4 of the 6 sites that block an AI crawler sitewide name that crawler explicitly, and 2 of those 4 are running Cloudflare's managed content-signals block — a 2026 policy someone chose, not a 2023 rule nobody re-read. The accidental blocks are the remaining ones, and they are a small number. The larger gap in this sample is not sites shutting AI crawlers out; it is the 8 sites whose robots.txt has nothing an AI crawler can read at all. Reading your own file takes 2 minutes; the free report reads it for you, names each of the 6 crawlers, and shows exactly which rule applies to each — so the decision on record is the one you actually made.