WordPress Discourage Search Engines Setting: What It Changes

The "Discourage search engines from indexing this site" checkbox in WordPress puts a robots meta tag with noindex, nofollow on every public page of the site. Google honors that tag: each page it recrawls with the tag leaves its search results, and new pages are not added. On a private staging copy that is the intended result. On a live site you want in Google, clear it now: it removes the whole site from Google one recrawl at a time, while nothing looks different to visitors.

What the setting outputs in current WordPress

The checkbox sits under Settings → Reading, in the row labelled "Search engine visibility", and the WordPress installer offers the same checkbox on its setup screen. It stores one value in the database: the blog_public option, 0 when the box is ticked and 1 when it is clear. We checked what that value changes against the WordPress core source in October 2026, when the current release was 7.1.3:

• Every front-end page gets a robots meta tag. The theme's standard head output includes this line:

<meta name='robots' content='noindex, nofollow' />

• The built-in XML sitemap switches off. The core sitemap at /wp-sitemap.xml returns a 404, and the Sitemap: line disappears from the robots.txt that WordPress generates.
• robots.txt does not block anything new. The generated file keeps its usual Disallow for the admin area and Allow for admin-ajax.php.
• No HTTP header is added. Core sends no X-Robots-Tag header for this setting, so the signal lives only in the HTML.
• The site stays public. The text under the checkbox reads "It is up to search engines to honor this request." Visitors, and any crawler that ignores robots meta tags, can still read every page.
• Administrators get one reminder. The "At a Glance" dashboard widget shows the line "Search engines discouraged" while the box is ticked.

If your dashboard is in Spanish, the wording depends on the locale. The WordPress translation project, checked in October 2026, renders the label as "Disuade a los motores de búsqueda de indexar este sitio" for Mexico, Argentina, Colombia and several other Latin American locales, and as "Pedir a los motores de búsqueda que no indexen este sitio" for Spanish as used in Spain. In most Spanish locales the row is called "Visibilidad en los motores de búsqueda" on the "Ajustes de lectura" screen, and in all of them the dashboard line reads "Motores de búsqueda disuadidos".

How the setting's behavior changed across WordPress versions

Advice written for older versions still circulates and describes a different mechanism. The core source shows 3 stages:

The 5.3 change matters most. Before it, the generated robots.txt told every crawler to stay out of the whole site, so Google would not fetch the pages and never saw their noindex tag. URLs that other sites linked to could stay in Google's index without being crawled, which Search Console reports as "Indexed, though blocked by robots.txt"; that status page explains the conflict. Since 5.3, the checkbox leaves robots.txt alone and relies on the meta tag alone. On a current site, a Disallow: / comes from a physical robots.txt file, a plugin or the host, never from this checkbox.

Why the setting ends up on a live site

The usual sequence is a staging site that was correctly hidden, followed by a launch that copies its database. Migration plugins, a host's "push to live" tools and a manual export and import all move the options table, and blog_public travels with it. The only visible change is a line in the page head that no visitor reads.

sequenceDiagram
    participant S as Staging site
    participant P as Production site
    participant G as Googlebot
    S->>S: Box ticked, blog_public is 0
    S->>P: Database copied at launch
    P->>P: blog_public is still 0
    G->>P: Recrawls the homepage and posts
    P-->>G: HTML with noindex, nofollow
    G->>G: Drops each recrawled page from results

In 45 external websites SEOReport audited between May and October 2026, none served a homepage carrying a noindex directive, either in the HTML or in an X-Robots-Tag header. A sample that size cannot say how common the mistake is, only that it is not routine. When it does happen, it applies to every public page at once, which is why it belongs on every launch checklist.

How to check the setting from outside the dashboard

What counts is the HTML a logged-out visitor receives after any page cache or CDN, because that is what Google fetches. Replace example.com with your domain, and repeat the first 2 commands for 2 or 3 posts as well as the homepage. The -L option follows redirects, so a move from the bare domain to www or to HTTPS does not hide the final page.

# 1. The robots meta tag in the served HTML
curl -sL https://example.com/ | grep -io "<meta[^>]*name=.robots[^>]*>"

# 2. Any noindex sent as an HTTP header instead
curl -sL -o /dev/null -D - https://example.com/ | grep -i "x-robots-tag"

# 3. What robots.txt actually serves
curl -sL https://example.com/robots.txt

# 4. Whether the core sitemap is switched off
curl -sL -o /dev/null -w "%{http_code}\n" https://example.com/wp-sitemap.xml

A noindex in the first result together with a 404 from the fourth matches the setting. An SEO plugin can replace the core sitemap with its own, so a 404 alone proves nothing. A noindex in the second result is a header rule from the server, host, CDN or a plugin, and the checkbox cannot remove it.

From inside WordPress, the fastest confirmation is the "Search engines discouraged" line on the dashboard. With command-line access, wp option get blog_public prints 0 when the box is ticked and 1 when it is clear.

flowchart TD
    A["Live HTML has a robots meta noindex?"] -->|yes| B["Is blog_public 0?"]
    A -->|no| C["X-Robots-Tag header has noindex?"]
    C -->|yes| D["Remove the server, host or CDN rule"]
    C -->|no| E["Live page is indexable: request a recrawl"]
    B -->|yes| F["Clear the checkbox and purge caches"]
    B -->|no| G["A plugin or theme adds noindex: check its settings"]
    F --> H["Live HTML clean on homepage and posts?"]
    G --> H
    H -->|no| I["A cache still serves old HTML: purge it"]
    I --> H
    H -->|yes| J["URL Inspection live test, then Validate fix"]

How the setting shows up in Search Console

As Google recrawls pages and finds the tag, they move to the "Not indexed" side of the Page Indexing report under the reason "URL marked 'noindex'". Google's Page Indexing report documentation, checked in October 2026, says the page was not indexed because Google found a noindex directive when it tried to index it. The move is gradual, because Google recrawls pages at different times. Our guide to the "URL marked 'noindex'" status covers the other causes of that status and when it is intentional.

URL Inspection shows both sides. The default view describes the version Google last indexed, and its "Indexing allowed?" field reports whether that version disallowed indexing. "Test live URL" fetches the page as it is now. After a fix, trust the live test; the indexed view changes only once Google recrawls.

A site launched with the box ticked from day 1 can show very few URLs in the report at all. The core sitemap is switched off, and nofollow asks Google not to follow the links on each page, so Google has little to discover from.

Plugins and server rules that change the picture

SEO plugins manage the robots tag themselves. We read the source of 2 widely used ones, Yoast SEO and Rank Math, in October 2026: both still force noindex while the checkbox is ticked, and both raise a warning in the admin area. With either plugin active, clearing the checkbox remains the fix.

The same plugins can keep a noindex in place after the checkbox is clear, through their own settings for individual posts, post types or archives. Any other plugin or theme can add directives through the same WordPress robots filter, for example a coming-soon or maintenance-mode plugin left active after launch. When the live HTML still shows noindex with blog_public at 1, those settings are where to look.

2 more leftovers come from staging. A physical robots.txt file with Disallow: / overrides the generated one, so re-read the live file after launch, especially if you saved robots.txt rules through a plugin while the site was hidden. A header rule that adds X-Robots-Tag: noindex works like the meta tag but never appears in the page source, which is why the outside check reads both. Our robots.txt and crawl access guide covers how to read the file line by line.

How to recover and how long it takes

1. Clear the checkbox and save, or run wp option update blog_public 1.
2. Purge every cache layer: the page-cache plugin, the host's cache and the CDN. A cached page keeps serving the old head until it expires.
3. Re-run the outside check on the homepage and a few posts, and confirm /wp-sitemap.xml or your plugin's sitemap now answers 200.
4. Submit the sitemap in the Search Console Sitemaps report, so Google has a full list of URLs to recrawl. Our XML sitemaps guide covers what a clean sitemap should contain.
5. Run a live test in URL Inspection on the homepage and your most important pages, then use "Request indexing" for that handful. The tool has a quota for individual URLs, and Google's documentation says repeated requests for the same URL do not speed up crawling.
6. Open "URL marked 'noindex'" in the Page Indexing report and select "Validate fix".

Then wait. Google's recrawl documentation, checked in October 2026, says crawling can take anywhere from a few days to a few weeks, and its noindex documentation notes that a revisit can take months for some pages. The Page Indexing report documentation says validation typically takes up to about 2 weeks and in some cases much longer. Indexing itself stays Google's decision: the URL Inspection documentation states that a request does not guarantee a page appears in the index.

How to keep it from happening at the next launch

Protect staging with a password, through HTTP authentication at the server or host, instead of relying on the checkbox. A login stops every visitor and crawler. Leave the checkbox clear on the password-protected copy: Google cannot read a page behind a login, so the box adds nothing there, and a database copied to production then carries no noindex with it. Then add one line to the launch checklist: run the outside check on the production domain after go-live and after every database migration or restore.

A full SEOReport diagnosis reads the robots meta tag and X-Robots-Tag header on your homepage, and the robots meta tag on the pages your sitemap lists, and names each page that tells search engines not to index it. A noindex on the homepage, which this setting always produces, is reported as a critical finding, so the setting is caught even when it has also switched the sitemap off. For a site that changes hands or hosting often, weekly monitoring, set up through the REST API, re-audits the site and emails you when a new critical finding appears. The broader pattern of sites hiding their own pages is covered in our indexability conflicts data and the page indexability guide.

Common questions

Does it block AI crawlers?

No. It adds no robots.txt rules, and robots.txt is where access for AI crawlers is usually controlled. It does reach Google's own AI features: Google's AI features documentation, checked in October 2026, says a page must be indexed to be shown as a supporting link in AI Overviews or AI Mode, so a noindexed page drops out of those too. Treat the checkbox as a search indexing request, not an AI policy. Our decision table for AI crawlers in robots.txt covers how to set one deliberately.

I cleared the box weeks ago and pages still show "URL marked 'noindex'". Why?

Search Console reports what Google saw at its last crawl of each page. Run "Test live URL" first: if the live test still finds a noindex, a cache, plugin or header rule is serving it. If the live test is clean, the page is waiting for a recrawl, and "Validate fix" plus a submitted sitemap is the right next step.

Is it safe to tick the box while I redesign a live site?

No. The setting applies to the live site, and every page Google recrawls during the redesign drops out of its results. Build the redesign on a password-protected staging copy instead.