Indexability Conflicts: When Your Own Site Tells Google to Go Away

Every audit we run asks a blunt question about the pages a site declares important — the homepage plus the URLs its own sitemap lists: can search engines index them as declared? Between May 5 and August 15, 2026, 21 of 49 audited sites failed that question. That is 42.9% of the sample telling crawlers to skip pages someone went to the trouble of publishing, linking, and listing.

The failures share a signature: every one of these sites works perfectly for humans. Pages render, links resolve, nothing in the browser hints that a directive underneath is sending search engines away. Indexability conflicts are the most self-inflicted category in our data — no competitor caused them, no algorithm update triggered them, and no one on the team can see them without fetching the site the way a bot does.

The dataset

Methodology: latest completed audit snapshot per domain from our current checkset, May 5 – August 15, 2026 — 49 audited sites, anonymized. Per-check denominators vary from 46 to 49 because a check only runs when its inputs exist — a site with no reachable sitemap yields no sitemap-page verdicts. The sample is self-selected — owners who ran an audit — and skews small-to-mid-size, so treat the rates as directional for that segment.

Redirects and canonicals cause more failures than noindex

The gap between the first 2 rows is the finding. The indexability check fails a page for any of 4 reasons: blocked by robots.txt, carrying a noindex directive, redirecting to a different URL, or declaring a canonical that points somewhere else. The noindex-specific check failed 7 sites — which means on most of the 21 failing sites, the non-indexable pages carry no noindex at all. They redirect away from the URL the sitemap promised, or they tell Google their canonical lives at a different address.

That distribution matters because teams hunt for the wrong culprit. The word everyone knows is noindex, so that is what gets grepped for — and it comes back clean. The actual conflict is usually structural: a sitemap generated from the CMS database while the live URLs migrated to a new path scheme, or a canonical tag templated to a URL variant that no page actually serves.

The check is careful about what counts. It excuses the redirect that is correct infrastructure — a naked domain hopping to its www variant — and it skips utility paths like /login and /signup that should carry noindex. The 21 failures are what remains after the benign cases are removed.

The 4 conflict patterns in the data

The staging noindex that shipped. 7 sites list pages in their sitemap that carry a noindex directive — in the meta robots tag or the X-Robots-Tag response header. This is the classic launch leftover: the directive that correctly hid the staging environment travels into production inside a template, a plugin setting, or a platform toggle. WordPress's "Discourage search engines from indexing this site" checkbox is the canonical example — a single setting, sitewide noindex, nothing visibly different.

robots.txt shouting over everything. 3 sites serve a robots.txt with Disallow: / for all bots — the site-wide go-away sign — and 5 fail the broader consistency check, where robots.txt blocking or a homepage noindex contradicts the evident intent to rank. The subtle trap in this pattern: robots.txt and noindex do opposite jobs, and combining them cancels the stronger one. A page blocked by robots.txt cannot be crawled, so a noindex placed on it is never read — which is how URLs end up in the "Indexed, though blocked by robots.txt" state, present in results as a bare link Google was forbidden to fetch.

Canonical contradictions. A canonical tag pointing at a target that itself carries noindex hands Google 2 instructions that cannot both be honored: consolidate signals into this page, and keep this page out of the index. Our engine resolves every canonical target and fails the audit when the target is noindexed, redirects, or refuses to declare itself canonical. The same contradiction appears within a single page when it carries both noindex and a canonical to somewhere else — asking Google to transfer authority through a page it was told to forget.

The sitemap promoting what directives forbid. A sitemap is a machine-readable claim that every URL in it deserves indexing. Listing a URL that robots.txt blocks or a directive noindexes makes the site argue with itself, and the argument costs crawl budget on every cycle. We measured this pattern on its own in our sitemap hygiene report.

Resolve conflicts in this order: intent, then a single mechanism, then proof

Indexability conflicts persist because fixes get applied signal by signal — someone patches a noindex here, edits robots.txt there — without anyone deciding what each page class is actually for. The durable fix runs in a single direction.

1. Decide intent per page class. Rankable content, duplicate variants, private utility pages, and infinite parameter spaces each get a single intent — before anyone touches a config file.

2. Express each intent through a single mechanism.

• Rank: listed in the sitemap, canonical pointing at itself, no robots directives at all.
• Consolidate duplicates: <link rel="canonical" href="https://example.com/primary/" /> on the variant, which stays crawlable and leaves the sitemap. Canonicals are a hint — Google's own documentation says it can pick a different canonical when other signals disagree, which is exactly why the target must be clean: indexable, 200, self-canonical.
• Keep out of results: a noindex meta robots tag in the page, or the X-Robots-Tag: noindex response header for PDFs and other non-HTML responses. The page must stay crawlable — a directive behind a robots.txt block is a directive that does not exist.
• Save crawl budget: Disallow: /search/ under User-agent: * in robots.txt, reserved for spaces with unbounded URLs. robots.txt controls crawling, never indexing — it removes nothing that is already indexed.

Per platform, the noindex intent is a single line: robots: { index: false } in a Next.js route's metadata export, the "Search engine visibility" toggle in WordPress — checked deliberately per environment, never inherited from staging — or add_header X-Robots-Tag "noindex" always; scoped to a location block in nginx.

3. Verify as a bot, per page class. The browser proves nothing here; fetch the way crawlers do:

curl -sI -A "Googlebot" https://example.com/page/ | grep -i "x-robots-tag\|location"
curl -s -A "Googlebot" https://example.com/page/ | grep -i "robots\|canonical"

A representative URL per page class is enough, checked against the intent you decided in step 1. Search Console's URL Inspection gives the authoritative second opinion, including which canonical Google actually chose. Our audit runs the full loop on every report — robots.txt against sitemap membership, directives against canonicals, canonical targets resolved and verified — and the free report lists each conflicting pair with the page it found it on.

The 42.9% failure rate puts indexability conflicts among the most common serious findings in our data, in the same tier as the header gaps from our most-failed checks ranking — with a harsher consequence, since a blocked page earns nothing regardless of how good it is. The sites that pass are the ones where indexing intent was decided once, written down in a single mechanism per page class, and checked the way a crawler would. Everything about this failure category is under the site owner's control — which is what makes it the most fixable 42.9% in the dataset.