URL Blocked by robots.txt: When It Is Right and When It Costs You

"URL blocked by robots.txt" means Google did not crawl the URL because a rule in your robots.txt file disallows it, and the URL is not in the index. On most sites the status is correct: admin areas, carts, internal search and parameter URLs are blocked on purpose. It costs you only when the rule catches a page you want in search, and the job is to find those pages among the intentional ones.

Until 2022 the Index Coverage report in Search Console used 2 labels for it: "Submitted URL blocked by robots.txt" for URLs listed in a sitemap you submitted, and "Blocked by robots.txt" for the rest. In German, Google's help documentation now lists the status as "URL wird von der robots.txt-Datei blockiert"; the older label was "Durch robots.txt-Datei blockiert". If your report says "Indexiert, obwohl durch robots.txt-Datei blockiert" ("Indexed, though blocked by robots.txt"), the URL is in the index despite the block, and that status has its own guide.

What Google says this status means

Google's Page Indexing report documentation (checked October 2026) lists "URL blocked by robots.txt" under "Not indexed" and says the page was blocked by your site's robots.txt file. The same documentation introduces the whole "Not indexed" group by saying these pages are not indexed "but not necessarily because of an error."

The block has 2 consequences for your decision:

• The block stops crawling, not indexing. Google's documentation adds that a block does not guarantee the page stays out of the index. If Google finds information about the page without loading it, for example links from other pages, there is a small chance it is indexed anyway and moves to "Indexed, though blocked by robots.txt".
• Google never sees the page. No title, no content, no links on it, no canonical, no noindex. Whatever the page says about itself is invisible while the rule stands.

Is it a problem? Sort intentional blocks from accidental ones

Start with the filter at the top of the Page Indexing report. Switch it from "All known pages" to "All submitted pages". Blocked URLs that appear under "All submitted pages" are listed in a sitemap you submitted: you are asking Google to index them while robots.txt forbids the crawl. That contradiction is almost always a mistake, whatever the rule was meant to do.

In 29 external websites submitted for an SEOReport audit whose sitemap pages we fetched between May and October 2026, 2 (7%) listed URLs in their sitemap that their own robots.txt disallows. It is not the most common sitemap problem, but it is the one that blocks pages the site has already declared important.

The accidental patterns, with before and after

Each pattern below is a rule that worked as written and blocked more than intended.

1. The staging file that shipped. A development site is blocked completely, and the file travels to production with the rest of the build.

# Before: copied from staging
User-agent: *
Disallow: /

# After: block only what should never be crawled
User-agent: *
Disallow: /cart/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

2. A prefix that is too short. Google matches a rule from the start of the URL path, so Disallow: /search blocks /search?q=shoes, and also /search-engine-basics/, a real article.

# Before
Disallow: /search

# After: block only the search results
Disallow: /search/
Disallow: /search?

3. A wildcard that catches landing pages. * matches any sequence of characters, and Google's matching includes the query string. Disallow: /*? blocks every URL with a parameter, including paginated category pages such as /shoes/?page=2 that you may want crawled so the products on them are found.

# Before
Disallow: /*?

# After: keep pagination crawlable, block the rest
Disallow: /*?
Allow: /*?page=

The same trap applies to file extensions. On a site whose pages end in .php, Disallow: /*.php blocks the whole catalog.

4. A forgotten Googlebot group. Google's robots.txt specification says a crawler follows only the most specific group that names it, and that specific groups and the * group are not combined. A User-agent: Googlebot group written for a migration years ago keeps applying to Googlebot, while the team edits only the * group and cannot explain the block.

# Before: Googlebot reads only its own group
User-agent: Googlebot
Disallow: /blog/

User-agent: *
Disallow: /cart/

The fix is to delete the old group or copy the current rules into it. The same rule explains a common side effect of AI crawler policies: a site that adds named groups for specific bots must repeat every shared rule in each of them. Our decision table for robots.txt and AI crawlers shows patterns that keep search and AI rules apart.

5. Blocked rendering resources. Disallow: /assets/ or Disallow: /static/ blocks the CSS and JavaScript your pages load. Google's robots.txt introduction says you can block unimportant resources, but not ones whose absence makes the page harder for Google to understand. Blocked files like these rarely show up as pages in the Page Indexing report, so look for them from the page side: run a live test in URL Inspection and open "View tested page", then "More info", to see the page resources Google loaded. Check the asset host too: robots.txt applies per host, protocol and port, so files served from cdn.example.com follow that host's own robots.txt. A blocked script that builds the page content matters most on sites that render in the browser, which our JavaScript rendering guide covers.

How to test a robots.txt rule against a URL

You do not need a tool to predict Google's decision. Google's specification makes it 4 steps.

flowchart TD
    A["URL to test"] --> B["Open robots.txt on the same host and protocol"]
    B --> C{"Is there a group for Googlebot?"}
    C -->|yes| D["Use only that group"]
    C -->|no| E["Use the * group"]
    D --> F["List every Allow and Disallow that matches the path"]
    E --> F
    F --> G{"Any match?"}
    G -->|no| H["Allowed"]
    G -->|yes| I["Longest rule path wins"]
    I --> J{"Allow and Disallow equally long?"}
    J -->|yes| H
    J -->|no| K["Apply the longest rule"]

1. Pick the file. Open /robots.txt on the exact host and protocol of the URL. https://www.example.com/robots.txt does not govern https://shop.example.com/.
2. Pick the group. Use the most specific User-agent group that matches Googlebot. User-agent names are not case-sensitive.
3. Find matching rules. A rule matches if its path matches the start of the URL path and query. * matches any characters, $ marks the end of the URL, and paths are case-sensitive: /Shop/ does not match /shop/.
4. Pick the winner. The rule with the longest path wins. If an Allow and a Disallow tie, Google uses the less restrictive one, the Allow.

Here is the method applied to the "after" file from pattern 3, plus Disallow: /search:

To check Google's own reading rather than yours, run URL Inspection on the URL. In the crawl details, "Crawl allowed?" shows whether a robots.txt rule blocks Google, and a live test applies the file Google can fetch right now. Google also publishes its robots.txt parser as an open-source library, which developers can run against a draft file before it ships.

Fix the rule, then verify

Change the file so the pages you want crawled are no longer matched. Narrow a broad Disallow, add a longer Allow for the exception, or remove a leftover group. Then:

1. Fetch the live file to confirm the server serves your edit, not a cached or build-time copy: curl -s https://www.example.com/robots.txt.
2. Request a recrawl of robots.txt from the robots.txt report in Search Console. Google generally caches the file for up to 24 hours, and the report's help names unblocking important URLs as a reason to request a recrawl. The report shows when Google last fetched the file and any errors or warnings; a line with an error is ignored.
3. Run a live test in URL Inspection on 2 or 3 affected URLs. "Crawl allowed?" should now say yes. Use "Request indexing" on the pages that matter most; Google's help says indexing can take up to a week or two.
4. Click "Validate fix" on the issue in the Page Indexing report once the rule is corrected. Google's documentation says validation typically takes up to about 2 weeks and can take much longer.

Google's help on recrawling says crawling can take anywhere from a few days to a few weeks, so expect the count to fall gradually. URLs that stay in the report afterwards should be the ones you meant to block.

Before you unblock a page that has been hidden for months, look at it the way a crawler will: its status code, its robots meta tag and its canonical. A page nobody could crawl may carry a forgotten noindex, which would move it straight to "URL marked 'noindex'". Our analysis of indexability conflicts covers how these signals interact.

Search Console tells you which URLs are blocked. A full SEOReport diagnosis checks the pages your sitemap lists against your robots.txt rules and names the ones you are asking search engines to find while telling them not to crawl. It also flags a site-wide disallow and a missing robots.txt. It reads the rules written for all crawlers, so a Googlebot-only group is still something to confirm in URL Inspection.

Common questions

Does a large number of blocked URLs hurt my site?

No, not by itself. Thousands of blocked parameter or search URLs are a sign robots.txt is doing its job. Count the blocked URLs your sitemap lists and the blocked pages you want ranked; those are the only ones that cost you.

What happens if my robots.txt is missing or returns an error?

Google's specification treats a 404 or other 4xx response (except 429) as if there were no robots.txt, so nothing is blocked. A 5xx error is different: Google stops crawling the site for the first 12 hours, then uses the last good copy it has for up to 30 days. Google treats timeouts and DNS errors on the file the same way, so a robots.txt that times out can pause crawling of a site that has no blocking rules at all.

Should blocked URLs be removed from my sitemap?

Yes. A sitemap lists the URLs you want indexed, so it should contain no URL that robots.txt blocks. Remove intentional blocks from the sitemap, and unblock the accidental ones. Our XML sitemaps guide covers what else belongs in the file.

Can I use robots.txt to remove a page from Google?

No. robots.txt controls crawling, and Google can still index a blocked URL that other pages link to. To keep a page out of search, let Google crawl it and add noindex. The robots.txt and crawl access guide explains which tool does which job.

Does a robots.txt on a subdomain or on HTTP affect my main site?

No. Each combination of host, protocol and port has its own file, and Google does not read robots.txt files placed in subdirectories. A blocking file on staging.example.com cannot block www.example.com, but the robots.txt on your asset or CDN host does decide whether Google can fetch the files your pages load.