Claude-User: the agent Claude sends when someone asks

Operator

Anthropic

robots.txt token

Claude-User

User agent

Not published by the operator

Purpose

Fetches a page when a user asks

Honours robots.txt

Yes, its operator states it does

Published IP ranges

https://claude.com/crawling/bots.json

Reverse DNS

None documented

Official documentation

Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?

Verified on

2026-10-05

What Claude-User does

Anthropic says that when individuals ask Claude questions, it may access websites using the Claude-User agent. [1]

Anthropic names the robots.txt token but does not publish a full user-agent string. [1]

Allow or block Claude-User in robots.txt

Add one of these groups to the robots.txt at the root of each host (each scheme and port counts separately) it should apply to. A crawler that finds one or more groups naming its token follows those groups, merged into one, and ignores the User-agent: * group, so repeat in them any * rules you still want applied (RFC 9309, section 2.2.1).

User-agent: Claude-User
Allow: /

User-agent: Claude-User
Disallow: /

What blocking Claude-User changes

Anthropic presents Claude-User as the control over these user-initiated requests: disabling it prevents Claude from retrieving a site's content in response to a user query, which may reduce the site's visibility for user-directed web search. [1]

Anthropic's robots.txt statement covers the bots it lists, Claude-User included. OpenAI and Perplexity say otherwise for their user-triggered agents. OpenAI says robots.txt rules may not apply to ChatGPT-User. Perplexity says Perplexity-User controls which sites user requests can access, and also that the fetcher generally ignores robots.txt rules. [1][3][4]

How to verify a request really is Claude-User

Anthropic publishes a list of IP addresses and says that a crawler whose source address is on it is coming from Anthropic. It also warns that blocking those addresses may not work correctly or persistently guarantee an opt-out, because doing so impedes Anthropic's ability to read robots.txt, and says opting out requires the robots.txt rule. [1][2]

A crawler policy is one part of being readable by assistants. The AI search readiness guide covers the rest, and robots.txt and crawl access covers the firewall and bot-protection rules that block crawlers before robots.txt is ever read. A full SEO report covers AI search readiness alongside the rest of a site's technical health.