AI Search Readiness: Being Usable by Assistants as Well as Crawlers
GPTBot, ClaudeBot and PerplexityBot read robots.txt independently. How to set an explicit crawler policy and publish an llms.txt worth following.
Assistants answer questions by reading pages directly and quoting them back, rather than by returning a ranked list of links. They reach your site through named crawlers, among them GPTBot, ClaudeBot, PerplexityBot and Google-Extended, and each one reads robots.txt independently. A site can be wide open to Googlebot and closed to every assistant, usually because a default rule was copied from a template. The first decision is a policy one, and it is yours: which agents may read your content, and for what. Making that explicit in robots.txt beats leaving it to a default, in either direction. An explicit allow tells an assistant it is welcome; an explicit disallow is a clear signal rather than an ambiguous silence. The llms.txt file is the second layer. It is a short Markdown document at your site root naming what the site is and linking to the pages you most want cited. It complements sitemaps and structured data, and it earns nothing on its own: the pages it links to still have to be reachable, current and worth quoting. These checks read your robots.txt from an assistant's perspective, then fetch llms.txt and follow every same-origin link inside it. A file naming pages that no longer resolve costs more credibility than having no file at all.
None of this has to be checked by hand. Every check on this page also runs on demand and returns the same crawler-policy, llms.txt and reference-resolution results as JSON, through the SEO report API and MCP server.
AI crawler rules missing in robots.txt
Why it matters
Explicit AI crawler rules reduce ambiguity and make it clearer which AI search systems may use your public content.
How we check it
robots.txt was scanned for explicit AI crawler Allow/Disallow rules.
How to fix it
Set an explicit robots.txt policy for the AI search crawlers you intend to serve, such as OAI-SearchBot, and verify that CDN or WAF rules allow the same requests. OAI-SearchBot access is required for page content to be considered for ChatGPT search summaries and snippets, but access does not guarantee appearance or placement.
AI crawlers blocked
Why it matters
If AI crawlers are blocked, your content is less likely to appear in AI-generated answers even when it is the best source.
How we check it
robots.txt was scanned for AI crawler disallow rules.
How to fix it
Remove Disallow rules that block AI search crawlers from accessing your content.
llms.txt missing
Why it matters
llms.txt gives AI agents a concise, owner-curated map of the site's purpose and preferred resources, reducing the work needed to extract structure from complex HTML. It supports inference-time context discovery; it does not guarantee rankings or model training inclusion.
How we check it
llms.txt was checked for presence at the well-known location.
How to fix it
Publish a factual /llms.txt Markdown file at the site root with an H1 site name, concise purpose summary, and clearly described links to preferred resources. Keep the linked pages crawlable and current; llms.txt complements rather than replaces sitemaps, structured data, and internal links, and does not guarantee rankings or training inclusion.
llms.txt format invalid
Why it matters
AI search tools are increasingly selective about what they can read and reference.
How we check it
llms.txt was validated against the llmstxt.org format specification.
How to fix it
Fix formatting issues in your llms.txt file. Follow the llmstxt.org specification.
llms.txt links broken
How we check it
Same-origin links in llms.txt were fetched and verified.
How to fix it
Ensure all URLs referenced in llms.txt resolve to valid pages on your site.
AI readiness gaps detected
How we check it
The report combined observed AI crawler policy, llms.txt availability and validity, and preferred-resource link checks without inferring rankings or model inclusion.
How to fix it
Address the listed AI-readiness gaps independently: publish explicit crawler policy in robots.txt, provide an owner-curated /llms.txt overview, and keep its preferred-resource links resolvable. robots.txt controls crawler access; llms.txt is an emerging inference-time context convention and does not guarantee discovery, citation, rankings, or model training inclusion.
See How Your Site Ranks
Get a free AI-powered SEO report with actionable findings and priority fixes for your website.
No signup required.