Claude-SearchBot: Anthropic's search crawler and its robots.txt rule

Operator

Anthropic

robots.txt token

Claude-SearchBot

User agent

Not published by the operator

Purpose

AI search indexing

Honours robots.txt

Yes, its operator states it does

Published IP ranges

https://claude.com/crawling/bots.json

Reverse DNS

None documented

Official documentation

Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?

Verified on

2026-10-05

What Claude-SearchBot does

Anthropic says Claude-SearchBot navigates the web to improve search result quality for users, analysing online content to make search responses more relevant and accurate. [1]

Anthropic names the robots.txt token but does not publish a full user-agent string. [1]

Allow or block Claude-SearchBot in robots.txt

Add one of these groups to the robots.txt at the root of each host (each scheme and port counts separately) it should apply to. A crawler that finds one or more groups naming its token follows those groups, merged into one, and ignores the User-agent: * group, so repeat in them any * rules you still want applied (RFC 9309, section 2.2.1).

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-SearchBot
Disallow: /

What blocking Claude-SearchBot changes

Anthropic states that disabling Claude-SearchBot prevents its system from indexing a site's content for search optimization, which may reduce the site's visibility and accuracy in user search results. [1]

Training is ClaudeBot's token. Anthropic presents its bots as separate so that owners can enable those they want and limit those they don't: a site can allow Claude-SearchBot while disallowing ClaudeBot. [1]

How to verify a request really is Claude-SearchBot

Anthropic publishes a list of IP addresses and says that a crawler whose source address is on it is coming from Anthropic. It also warns that blocking those addresses may not work correctly or persistently guarantee an opt-out, because doing so impedes Anthropic's ability to read robots.txt, and says opting out requires the robots.txt rule. [1][2]

A crawler policy is one part of being readable by assistants. The AI search readiness guide covers the rest, and robots.txt and crawl access covers the firewall and bot-protection rules that block crawlers before robots.txt is ever read. A full SEO report covers AI search readiness alongside the rest of a site's technical health.