AI Agent SEO Check: What the Agent Actually Sees on Your Page

When you ask an AI agent to check a page's SEO, it usually works from the page converted to text. The headings and copy reach it; the HTTP status, response headers, redirect hops, head tags and structured data usually do not. The agent can therefore review your writing well, and it cannot confirm most of what decides whether a search engine indexes the page.

Fetch tools differ between assistants and change over time, so the useful question is what the agent received. In October 2026 we answered it for one of our own pages, recording the full HTTP exchange next to the text view an agent works from. This article reports exactly what we ran, what each view contained, and how to give an agent the evidence it is missing.

We fetched one of our own pages both ways

We chose a published article, our redirect chains guide, and requested it by the long address a person might paste into a chat: http://www.seoreport.dev/articles/redirect-chains-guide/. Plain HTTP, the www host and a trailing slash are all variants the site normalizes, so the request was guaranteed to redirect.

We read it 3 ways:

1. The full HTTP exchange, with curl, recording every hop, the final status line, every response header and the raw HTML.
2. A plain text extraction of that same HTML: scripts, styles and the <head> removed, visible text kept. This is the conversion many fetch tools perform before an assistant reads a page.
3. The fetch tool built into one AI coding assistant, asked to report only what was literally present in the content it received, item by item.

Views 2 and 3 agreed on every item that matters here, so the table shows them together. The only difference: the fetch tool passed the page title through, while the plain extraction kept the H1, which carries the same words.

By size, the text view was about 10 KB of a 71 KB HTML response. Most of what was dropped is script and style code, which no SEO review needs. The rest is the evidence an SEO review exists to read.

Two details from the table matter more than the counts. First, the answer about the canonical tag depended on how we asked. Asked to mark each tag "present or absent", the fetch tool answered "absent" for a canonical the page has, and only added a caveat about the <head> afterwards. Given the option "cannot tell", it chose that instead. An agent that receives the first answer and acts on it would recommend adding a tag that already exists. Second, the Article JSON-LD and an article:modified_time tag both carry a modification date of October 3, 2026, while the visible byline shows only the publication date of September 28, 2026. The date a search engine can read as "last updated" exists only in the part of the page the agent never received.

Nothing on this page was wrong. Two redirect hops are well inside the 10 that Google's crawling documentation says its crawlers follow, and the canonical points to the page itself. The finding is that the agent could not have known either way.

The fetch tool reported a status only when the page failed

We repeated the test with an address that does not exist on our site. The server answered 404, and the fetch tool said so: it reported "HTTP 404 Not Found" and returned no page text. On the working page, the 200 was never mentioned.

That pattern (errors surface, successful responses stay silent) leaves out the cases that cause the most trouble, because they answer 200:

• A page that reads as missing but answers 200. Our 404 page says "Article Not Found. The article you are looking for does not exist." If a server sent that same text with a 200 status, the text view would be identical. Only the status line tells search engines which one it is, and when Google detects the 200 version, Search Console reports it as "Soft 404".
• A page that reads perfectly but tells search engines not to index it. Google's robots meta tag documentation (checked October 2026) states that any rule usable in a robots meta tag can also be sent as an X-Robots-Tag HTTP header. A noindex sent that way leaves no trace in the text. We cover how to find and decide on these in URL marked 'noindex' in Search Console.
• A page whose canonical is declared in a header. Google documents a Link response header with rel="canonical" as an alternative to the HTML tag, including for non-HTML files such as PDFs. A text view cannot see it.

The evidence lives in the exchange, not on the page

Everything the text view dropped belongs to one kind: evidence that a server and a crawler exchange, and that a visitor reading the page never sees. It falls into 6 places.

Status and redirects. Each hop is its own response with its own status code. Some fetch tools rewrite http:// to https:// before requesting, so the HTTP-to-HTTPS redirect is never requested at all. The redirects and URL normalization guide covers which hops to keep.

Response headers. X-Robots-Tag, Link, Cache-Control, Vary and the security headers all travel above the HTML. None of them is text on the page.

Head tags. The canonical, meta robots, hreflang, meta description and Open Graph tags sit in the <head>, which text conversions remove.

Structured data. JSON-LD lives inside a <script> element. Text conversions remove scripts, and the JSON-LD goes with them. The structured data and social metadata guide explains what it should contain.

Raw versus rendered HTML. Some fetch tools run JavaScript before converting the page, others do not, and the output rarely says which. A page whose content only appears after JavaScript runs can look complete to one tool and empty to another. We measured how often that gap appears in what AI crawlers actually see.

Files and pages beyond the URL. robots.txt is a separate file the agent never requests unless asked. Google's documentation notes that when robots.txt blocks a page, Google cannot read that page's noindex at all, so the two have to be checked together. The sitemap, the internal links and any duplicate URLs live on other pages entirely.

sequenceDiagram
    participant U as You
    participant A as AI agent
    participant F as Fetch tool
    participant S as Your site
    U->>A: Check the SEO of this page
    A->>F: Fetch the URL
    F->>S: GET http://www.example.com/page/
    S-->>F: 301 and a Location header
    F->>S: GET https://example.com/page/
    S-->>F: 301 and a Location header
    F->>S: GET https://example.com/page
    S-->>F: 200, response headers, full HTML
    Note over F: Hops, status, headers, head tags and JSON-LD dropped
    F-->>A: Visible text as Markdown
    A-->>U: Advice based on the text alone

Most SEO issues we flag are outside the page text

Our own audit data shows how much of a review depends on what the text view drops. Across 45 external websites SEOReport audited between May and October 2026, 94% of the 506 issues flagged were somewhere a reader of the page never sees: response headers, head tags, structured data, the XML sitemap, robots.txt, redirects, or relationships between pages.

The chart below breaks the same 506 issues down by the one place a person would have to look to see the evidence.

Only the 5.9% in visible page content is the kind of evidence a text view keeps. Grouped by what an agent would have to receive to see them, the other locations fall into 3 groups. HTTP status and headers, head tags and structured data add up to 69% of the 506 issues: evidence that arrives inside a page's own response and that a text conversion throws away. A better text extraction would not recover it; reading the response itself would. The XML sitemap, the article library read as a whole, relationships between pages and robots.txt add up to about 20%: evidence that no single page's response contains. The remaining 5.6% needs a second view of the page or of its address: the rendered page next to the raw HTML, the smartphone response next to the desktop one, or the redirect hops in front of the page.

The same sample shows the effect per site: every one of the 45 sites had at least one flagged issue outside the page's visible text, and for 24 of them (53%) every flagged issue was outside it. On those 24 sites, a review limited to visible text would have had none of the flagged evidence in front of it.

How to give your agent the evidence it is missing

An agent reasons well over evidence it can see. The fix is to put the exchange in front of it.

If your agent has a shell, ask it to run these commands and show you the raw output before it draws any conclusion. If it does not, run them yourself and paste the output into the chat.

# 1. Every hop, every status, the headers that matter (GET, following redirects)
curl -sL -D - -o /dev/null http://www.example.com/page/ \
  | grep -iE '^(HTTP/|location|x-robots-tag|link|content-type|cache-control|vary)'

# 2. Canonical, hreflang and meta robots tags from the raw HTML
curl -s https://example.com/page \
  | grep -oiE '<link[^>]*rel="(canonical|alternate)"[^>]*>|<meta[^>]*name="robots"[^>]*>'

# 3. How many JSON-LD blocks the raw HTML contains
curl -s https://example.com/page | grep -o 'application/ld+json' | wc -l

# 4. The crawl rules that apply to the page
curl -s https://example.com/robots.txt

Three habits make the answers reliable:

• Use GET, not HEAD. Some servers answer the two differently. Ours did: the first hop in our test answered 301 to a GET request and 308 to a HEAD request.
• Ask what it received before asking what is wrong. "List the status code, headers and head tags you can see" turns a hidden gap into a visible one, and shows you which "absent" answers mean "never received".
• Ask whether the HTML was rendered. If the agent's tool ran JavaScript, its view can differ from what the server sends. Compare it with the raw HTML from command 2, as the JavaScript rendering guide describes.

A complete check of one page is still one page

With the full exchange in front of it, an agent can review a single page properly. What the response cannot contain is the rest of the site: whether your sitemap lists the page, whether other pages link to it, whether another URL duplicates it, whether robots.txt blocks pages your sitemap advertises. In the sample above, about one in five flagged issues lived where no single page's response could show it. Covering those means repeating the work across many pages, in many sessions, with a method that stays the same each time, which is the subject of why one AI session cannot audit a whole website.

A full SEOReport diagnosis does much of that site-level work in one report. It requests the pages your sitemap lists, traces the address you submit through every redirect hop, reads your robots.txt, and compares your homepage's raw HTML with the rendered page. Each finding carries its evidence (status codes, header values, affected URLs), and each finding drawn from a sample of pages states how many it examined. An agent can create and read that report through the MCP server and REST API, so it starts from the site's evidence instead of from text.

Common questions

Can ChatGPT check my website's SEO?

It can review whatever content reaches it: headings, copy, and the title if the tool passes it through. Whether it also sees the status code, headers, head tags or structured data depends on the fetch tool it uses at that moment, which varies and is rarely stated. Ask it to list what it received first, or paste the output of the commands above.

Is an AI SEO checker the same as an SEO audit?

An AI SEO checker usually reviews one page from the content its tool returns. An audit reads the HTTP exchange, robots.txt, the sitemap and the relationships between pages, and records the evidence for each finding so the result can be checked and repeated. The two answer different questions: one reviews a page's writing, the other establishes what the site serves to crawlers.

Why did my agent say my page has no canonical tag when it does?

The canonical tag lives in the <head>, which text conversions remove. In our test the fetch tool answered "absent" for a canonical that is present. Confirm with command 2 above, or by viewing the page source in a browser, before acting on that kind of advice.

Does it matter whether my agent's fetch tool runs JavaScript?

Yes, when your content or tags depend on JavaScript. A tool that renders the page can report a title or H1 that the raw HTML lacks, and a tool that does not can report a near-empty page that browsers show in full. Comparing the raw HTML with the rendered page settles which one search engines and other crawlers receive first.