AI Search Readiness: Being Usable by Assistants as Well as Crawlers

Assistants answer questions by reading pages directly and quoting them back, rather than by returning a ranked list of links. They reach your site through named crawlers, and each one reads robots.txt independently. OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages for ChatGPT search, Claude and Perplexity answers, and Claude-User fetches a page when someone asks. GPTBot, ClaudeBot and CCBot collect training data instead, so blocking them is a licensing choice that does not take a site out of AI search. A site can be wide open to Googlebot and closed to every assistant, usually because a default rule was copied from a template. The first decision is a policy one, and it is yours: which agents may read your content, and for what. Making that explicit in robots.txt beats leaving it to a default, in either direction. An explicit allow tells an assistant it is welcome; an explicit disallow is a clear signal rather than an ambiguous silence. The llms.txt file is the second layer. It is a short Markdown document at your site root naming what the site is and linking to the pages you most want cited. It complements sitemaps and structured data, and it earns nothing on its own: the pages it links to still have to be reachable, current and worth quoting. Review both from an assistant's point of view: read your robots.txt as each named crawler would, then open llms.txt and confirm that every link inside it still resolves. A file naming pages that no longer resolve costs more credibility than having no file at all.

None of this has to be reviewed by hand. On a paid plan, the same crawler-policy and llms.txt findings are available on demand as JSON, through the SEO report API and MCP server.

SEOReport's paid diagnosis reviews this across the pages of your own site, shows the evidence behind every finding, and ranks the fixes by priority. See plans and pricing.