Robots.txt and Crawl Access: Making Sure Crawlers Can Reach You
Crawlers have to fetch a page before they can rank it. How robots.txt rules, bot protection and soft 404s block access while the site looks healthy.
Before a search engine can evaluate a page it has to fetch it. Crawl access is that first request, and it depends on 3 things agreeing: robots.txt permits the path, the server returns real HTML, and no protection layer intercepts the request on the way. Bot protection is the fastest-growing source of accidental blocking. A firewall rule tuned against scrapers regularly catches Googlebot, Bingbot and the AI crawlers alongside them. The site is perfectly usable in a browser while the crawler receives a challenge page, a 403, or a JavaScript interstitial. Because the response is technically successful, uptime monitoring stays green. Soft 404s cause the mirror-image problem: the server returns 200 alongside a page saying the content is missing, so search engines store an error state as content and the real page stops being crawled. Robots.txt itself deserves care. A single site-wide Disallow left over from a staging environment removes an entire domain from search. The file is fetched constantly and cached briefly, so a mistake spreads within hours and a fix propagates just as fast. These checks fetch your site the way an external crawler would, with no cookies, no warm cache and no browser fingerprint. What they see is what search engines see.
Robots.txt not found
Why it matters
Search engines need to be able to reach the pages you want appearing in results.
How we check it
The robots.txt file was fetched and parsed.
How to fix it
Create a robots.txt file at the root of your domain. Why this matters: robots.txt is the first file crawlers request; without it, they have no guidance on which sections to avoid, and some crawlers treat the absence as a signal of an unmaintained site.
Robots.txt blocks all bots
How we check it
robots.txt rules were checked for blanket 'Disallow: /' blocking directives.
How to fix it
Remove the 'Disallow: /' rule from robots.txt that blocks all user-agents.
Aggressive bot protection detected
How we check it
Branded HTTP responses and browser-rendered content were checked for explicit access-denial and challenge signatures.
How to fix it
Configure the WAF or hosting security layer so verified search crawlers and explicitly approved audit agents can retrieve public HTML, robots.txt, sitemap files, and llms.txt without a CAPTCHA. Preserve rate limits and abuse controls, and verify the result from an independent crawler network before closing the issue.
Homepage HTML not accessible
Why it matters
If search engines cannot reach your pages reliably, even strong content may not rank.
How we check it
The homepage Content-Type and response body were inspected.
How to fix it
Verify your homepage returns valid HTML content, not a redirect loop, error page, or blank response.
Homepage soft 404 detected
How we check it
Homepage content was checked for soft-404 patterns.
How to fix it
Fix the homepage to return meaningful content instead of a 200-status not-found page. Why this matters: A soft 404 wastes crawl budget and confuses search engines—they may continue indexing a page that offers no value to users.
Homepage challenge detected
How we check it
Homepage was checked for bot challenge or CAPTCHA patterns.
How to fix it
Remove crawler-facing challenges or CAPTCHAs from the homepage, or configure the WAF to allow verified search crawlers while preserving abuse controls. A challenge response can prevent a crawler from retrieving the page content, but this report does not infer downstream search outcomes.
See How Your Site Ranks
Get a free AI-powered SEO report with actionable findings and priority fixes for your website.
No signup required.