2 in 5 Sites We Audit Are Blocking at Least 1 AI Crawler

Every audit we run fetches robots.txt and reads it the way an AI crawler would. Between May 5 and August 15, 2026, 20 of the 49 sites we audited — 40.8% — served rules that shut out at least 1 of the 6 major AI crawlers we probe, across the entire site. Another 7 sites (14.3%) sat at the opposite extreme: no rules that apply to AI crawlers at all, not even a wildcard directive for them to obey.

That leaves under half the sample — 22 sites — with a robots.txt that states an explicit AI-crawler policy and blocks none of them sitewide. In a year when ChatGPT search, Perplexity, and Claude cite sources by fetching them, the file most owners have not opened since launch has quietly become a visibility switch.

Methodology: latest completed audit snapshot per domain from our current checkset, May 5 – August 15, 2026, anonymized. Per-check denominators vary between 46 and 49 because not every check runs on every site — these 2 checks require a successfully served robots.txt. The sample is self-selected — owners who ran an audit — and skews small-to-mid-size, so treat the rates as directional for the long tail of the web rather than the web at large.

Blocked in this data means a sitewide disallow in robots.txt

Our engine parses every robots.txt group and evaluates 6 user agents against it: GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Claude-SearchBot, and CCBot. A crawler counts as blocked only when a group matching it — by name, or through the User-agent: * wildcard — carries a sitewide Disallow: /. Partial disallows like Disallow: /admin register as an explicit policy, which is healthy, and they pass.

2 properties of that definition matter for reading the numbers. First, it is conservative: a WAF challenge, a CDN bot rule, or a firewall block never appears in robots.txt, so 40.8% is a floor on how many sites actually turn these crawlers away. Second, the wildcard clause means a site can block every AI crawler without ever typing an AI crawler's name — a blanket disallow written for a staging environment, or a template nobody has re-read in years, applies to GPTBot exactly as it applies to everything else.

Most of these blocks answer a question from 2023

The blocking pattern we see matches what larger external samples report. A March 2026 analysis of 10,000 sites by SEO Score Tools found 18.7% actively block GPTBot and 41.3% carry no AI-specific robots rules at all. Their no-rules figure lands higher than our 14.3% — definitions differ between samples, and our check counts a wildcard group as an applied policy — but both datasets agree on the shape: a large minority of the web has made an AI-access decision it never consciously took.

The unintentional part is the story. In 2023, when GPTBot first appeared in server logs and CCBot became newly notorious as a training source, the only known consequence of access was model training — so bot-block lists circulated, plugins shipped 1-click toggles, and thousands of robots.txt files inherited disallows that nobody has re-read since. The question those rules answered was "do I want my content in a training corpus?" The question that matters in 2026 is different: "do I want to be findable and citable when an assistant answers on my topic?" A rule written for the first question now silently answers the second.

Training crawlers and search crawlers deserve different answers

The 6 crawlers we probe split cleanly into 2 jobs, per each operator's published bot documentation:

• Search and retrieval. OAI-SearchBot indexes content so ChatGPT search can surface and link to it — OpenAI documents that this access, and only this access, governs whether pages are considered for ChatGPT search results. PerplexityBot builds Perplexity's search index. Claude-SearchBot indexes to improve Claude's search-backed answers. Blocking any of these removes you from that assistant's citations.
• Training. GPTBot gathers content that may train OpenAI's models. ClaudeBot does the equivalent crawling for Anthropic. CCBot feeds Common Crawl, the open corpus behind many research and training datasets. OpenAI is explicit that blocking GPTBot has no effect on ChatGPT search inclusion — the 2 pipelines are separate.

Google runs the same split under different names: its AI surfaces ride on ordinary Googlebot access, while the separate Google-Extended token controls Gemini training and grounding. We covered that machinery in our guide to Google's generative search.

The split turns a vague anxiety into a 2-part decision. Blocking training crawlers is a legitimate stance on how your content may be reused, and it costs you nothing in AI search. Blocking retrieval crawlers is a visibility decision — the same category as noindexing yourself — and it deserves the same deliberateness.

Write a robots.txt that records the decision

Here is a policy that stays fully visible in AI search while withholding training consent — the most common deliberate split we see:

# Search & retrieval — open, so assistants can find and cite this site
User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

# Training — withheld; this has no effect on AI search visibility
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

If you want everything open, say so explicitly rather than by omission — a named Allow: / group per crawler documents that someone decided, which is exactly what the 14.3% with no applicable rules are missing.

3 verification steps close the loop. Fetch https://yoursite.com/robots.txt from outside your network and read what production actually serves — CDNs and platforms can inject or override the file in your repository, and some edge providers ship 1-click AI-bot blocking that overrules your directives entirely. Check any WAF or bot-management rules for the same 6 user agents, because robots.txt permission means nothing to a crawler that gets a 403. Then re-run the check after every infrastructure change; a provider migration can flip this switch without touching your code.

1 note on how our audit reads the deliberate policy above: the sitewide disallows on GPTBot, ClaudeBot, and CCBot will still surface as a finding. That is by design. A sitewide AI-crawler block should always be an affirmed decision, and the finding is where you affirm it — the failure mode this check exists to catch is the block nobody remembers writing.

The cheapest visibility audit you will run this year

robots.txt is 1 of 2 files through which your site speaks to AI systems — the other is llms.txt, where our data shows 76% of files fail the agents they were written for. Both share the same failure signature as the blocks in this report: written once, syntactically plausible, never read back.

The sites in our sample with a clean AI-crawler posture were not the biggest or the best resourced. They were the sites where someone had opened the file since 2023 and answered the current question. Reading yours takes 2 minutes; the free report reads it for you, names each of the 6 crawlers, and shows exactly which rule applies to each — so the decision on record is the one you actually made.