robots.txt for AI Crawlers: Search, Training and User Fetches

AI crawlers serve 3 different purposes, and robots.txt controls each one through its own token: crawling for an AI search or answer feature, collecting pages for model training, and fetching a page because a person asked a question. Most sites want the first, make a deliberate choice about the second, and can only partly govern the third. A single "block AI" rule mixes them up, and the usual casualty is AI search visibility the owner meant to keep.

The table follows the operators' own documentation, checked October 2026 and tracked crawler by crawler in our AI crawler reference.

Which robots.txt token controls which AI purpose?

Every token is independent of the others. OpenAI states that each of its settings works on its own, so a site can disallow GPTBot and still allow OAI-SearchBot, the crawler behind ChatGPT search. When a site allows both, OpenAI says it may use 1 crawl for both uses.

3 rows need a closer reading.

Some training crawlers do more than training. Meta says Meta-ExternalAgent crawls for uses such as training foundation models or improving products by indexing content directly. Amazon says Amazonbot improves its products and services and may be used to train its AI models. Their search agents, Meta-WebIndexer and Amzn-SearchBot, have their own tokens, and Amazon says Amzn-SearchBot does not crawl content for generative AI training.

2 tokens never crawl. Google states that Google-Extended has no user agent of its own: Google's existing crawlers fetch the page, and the token decides whether the content may train future Gemini models and ground answers in Gemini Apps and in Grounding with Google Search on Vertex AI. You will never see Google-Extended in a server log. Apple says the same of Applebot-Extended: Applebot fetches, and the token only decides how Apple may use what it collected.

User-triggered agents sit partly outside robots.txt. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person initiated the request. Perplexity says Perplexity-User controls which sites user requests can access, and also that the fetcher generally ignores robots.txt rules. Meta and Amazon give similar warnings for Meta-ExternalFetcher and Amzn-User. Anthropic is the exception: its statement that its bots honor robots.txt covers Claude-User.

Which policy fits your site?

Answer the questions in order; each path ends at one of the patterns below.

flowchart TD
    A["Do you want to appear in AI search answers?"] -->|yes| B["Leave the AI search tokens under your normal rules"]
    A -->|no| C["Disallow the AI search tokens"]
    B --> D["Do you accept model training on your public pages?"]
    C --> D
    D -->|yes| E["No training groups needed"]
    D -->|no| F["Disallow GPTBot, ClaudeBot, CCBot, meta-externalagent and Amazonbot"]
    F --> G["Opt out of Google and Apple training too?"]
    G -->|yes| H["Add Google-Extended and Applebot-Extended"]
    G -->|no| I["Leave those 2 tokens out"]
    E --> J["Must user-triggered fetches be refused?"]
    H --> J
    I --> J
    J -->|yes| K["robots.txt alone cannot do it: enforce at the server or CDN"]
    J -->|no| L["Publish the file and verify it live"]

Why a named group can reopen paths your wildcard rules closed

Know this matching rule before copying any pattern. Google's robots.txt documentation and RFC 9309, the robots.txt standard, agree: a crawler obeys only the most specific group that names it and ignores every other group. A named group does not inherit the User-agent: * rules.

User-agent: *
Disallow: /cart/
Disallow: /account/

User-agent: OAI-SearchBot
Allow: /

With this file, OAI-SearchBot may crawl /cart/ and /account/, because its own group is the only one it reads. The well-meant Allow group removed the private-path rules for the one crawler it names. Either delete the named group, so the crawler falls back to *, or repeat the private paths inside it.

Other matching facts that change real files:

• Several User-agent lines can share one set of rules, which keeps a long list of AI tokens readable.
• If 2 groups name the same crawler, their rules combine. A named group is never combined with *.
• Token names match without regard to case; paths are case-sensitive, so Disallow: /Cart/ does not cover /cart/.
• The longest matching path wins. On a tie between Allow and Disallow, Google uses the least restrictive rule.
• A crawler with no named group and no * group has no rules at all.
• Each host serves its own file: www.example.com and blog.example.com need separate rules, and Anthropic asks owners to add the opt-out on every subdomain they want excluded.
• Apple says Applebot follows the Googlebot group when robots.txt names Googlebot but not Applebot. Apple does not say whether Applebot-Extended inherits rules that way, so name it explicitly.

Copy-paste robots.txt for the 3 common policies

Each pattern keeps an ordinary * group for private paths. Replace /cart/ and /account/ with your own, and the sitemap address with yours.

Allow AI search, opt out of training

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Disallow: /cart/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

The AI search crawlers have no named group, so they follow * and stay out of the private paths without any extra lines. That is the safest way to allow them. CCBot is on the list because Common Crawl provides its archive to researchers, companies and individuals for research and analysis, which includes training; drop it if you want your pages in that archive. If you want Meta's or Amazon's product uses, remove meta-externalagent or Amazonbot, for the reasons given after the table. Meta prints its tokens in lowercase; copy that spelling.

Allow all AI crawlers

User-agent: *
Disallow: /cart/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

No AI-specific lines are needed: every crawler without a named group follows *. A site with no robots.txt at all is also open, since a missing file means no rules, but it states no policy and names no sitemap. In 45 external websites SEOReport audited between May and October 2026, 3 (7%) had no robots.txt.

Block AI crawlers, keep conventional search

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: CCBot
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Disallow: /cart/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

3 limits apply. ChatGPT-User, Perplexity-User, meta-externalfetcher and Amzn-User may not follow their lines. Any agent you do not name, including ones introduced after you publish, follows *. And Google's AI Overviews and AI Mode are governed by Googlebot and Search snippet controls, not by any line in this file; the common questions below explain that case.

When a robots.txt rule is not enough

RFC 9309 states plainly that robots.txt rules are not a form of access authorization. A crawler that honors the file stays out; anything else can request the page, and any client can copy a crawler's name. If a refusal must hold, enforce it where requests arrive: the server, the CDN or the firewall. Most operators publish IP address lists for that check. Our reference names each operator's verification method, and notes where none exists, as with Meta-ExternalAgent.

The same layer causes the opposite failure. A firewall or bot-protection rule that challenges OAI-SearchBot undoes an Allow in robots.txt, and OpenAI recommends allowing requests from its published addresses for sites that want to appear in ChatGPT search. Our guide to checking whether bot protection blocks the AI search traffic you want walks through that comparison.

How to confirm the live file says what you meant

1. Fetch the file the way a crawler does. Run curl -s https://www.example.com/robots.txt and read the live body, not your repository copy. A hosting platform or CDN that manages robots.txt can serve something different. Check the status code too: Google treats a 4xx response other than 429 as if no robots.txt existed. After a 5xx, Google stops crawling the site for the first 12 hours, then uses its last good copy for up to 30 days.
2. Repeat on every host that serves pages: www, the apex and each subdomain.
3. Read each named group in isolation. For every token you listed, ask what that crawler may fetch if this group were the only one in the file. Private paths missing from a named group are open to that crawler.
4. Allow time before judging. OpenAI says a change takes about 24 hours to reach its systems; Perplexity and Meta say up to 24 hours. Google generally caches robots.txt for up to 24 hours. Amazon gives about 24 hours, and also says its agents may use a cached copy from the last 30 days.
5. Check your logs by token, then by address. Match on the token, such as GPTBot, rather than the full user agent string, and confirm the source address against the operator's published list.

robots.txt is a site-level file that sits outside every page, so a check that reads one page at a time does not see it; what an AI agent actually sees when it checks your page covers that gap. A full SEOReport diagnosis reads your robots.txt the way AI crawlers do and tells you whether the AI search crawlers can reach you, separately from any training opt-out you have chosen. It reads the rules for GPTBot, ClaudeBot, CCBot, OAI-SearchBot, Claude-SearchBot, PerplexityBot and Claude-User by name, and reports bot protection or challenges that would stop automated visitors. For how audited sites actually wrote these rules, see our AI crawler robots.txt audit data, and for robots.txt beyond AI agents, the robots.txt and crawl access guide.

Common questions

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI states that GPTBot and OAI-SearchBot are independent settings. GPTBot governs training; OAI-SearchBot decides whether your pages can be shown in ChatGPT search answers. Disallow GPTBot alone and ChatGPT search still has the access it needs.

Does blocking Google-Extended keep my pages out of AI Overviews?

No. Google's documentation on AI features and your website, checked October 2026, names robots.txt rules for Googlebot as the access control and lists nosnippet, data-nosnippet, max-snippet and noindex as the ways to limit what appears. A page must be indexed and eligible for a snippet to be a supporting link in AI Overviews or AI Mode. Google-Extended governs Gemini training and grounding, not Search features.

Does a training opt-out remove content that was already collected?

The operator pages we track describe what a rule signals from the time it is read. Anthropic, for example, says restricting ClaudeBot signals that the site's future materials should be excluded from training. None of them describes removing content collected earlier through robots.txt. Common Crawl separately keeps an Opt-Out Ledger of legal opt-out requests it has received.

Do I need a rule for every AI crawler?

No. Name only the agents you want to treat differently from *. Every unnamed crawler follows the * group. A long list of named groups is only safer when each group carries the private paths it needs.