Indexed, Though Blocked by robots.txt: The Fix

'Indexed, Though Blocked by robots.txt': What It Means & Fix

July 18, 20264 min read

Some links here are affiliate links: if you buy through them I may earn a commission, at no extra cost to you. I only recommend tools I'd point a colleague to, and rankings are never paid for.

“Indexed, though blocked by robots.txt” means Google added a URL to its index even though your robots.txt told it not to crawl the page — and the fix is almost never “add more robots.txt rules.” This one confuses everyone, because the solution is the opposite of what the warning seems to suggest. Here’s what’s really happening and how to fix it correctly.

Why this warning is a contradiction (that makes sense)

It sounds impossible: blocked and indexed? Here’s the key insight that resolves it:

robots.txt controls crawling, not indexing. Disallowing a URL tells Google “don’t read this page.” It does not say “don’t list this URL.” So if Google discovers the URL another way — usually because other pages link to it — it can index the URL based on those external signals (anchor text, the URL itself) without ever reading the page.

That’s exactly this status: Google found the URL, respected your “don’t crawl” rule, but indexed it anyway from the links pointing at it. The result is often an ugly search listing with no description (“No information is available for this page”), because Google was never allowed to read the content.

Why adding robots.txt rules makes it worse

Here’s the trap: people see “blocked by robots.txt” and think the block failed, so they add more disallow rules. That’s backwards. The block is why the page is stuck — because Google can’t crawl the page, it can never see a noindex tag on it, so it can never cleanly remove it. You’ve created a page Google can list but can’t evaluate. More robots.txt rules deepen the trap.

The correct fix

It depends on whether you want the page indexed:

If you want the page OUT of Google

Remove the Disallow rule for that URL in robots.txt, so Google can crawl the page again.
Add a noindex directive to the page — either a <meta name="robots" content="noindex"> tag in the HTML <head>, or an X-Robots-Tag: noindex HTTP header.
Let Google recrawl. Now it can read the page, see the noindex, and drop it from the index properly.
Once it’s gone, you can re-block it in robots.txt if you want to save crawl budget — but only after the noindex has done its job.

The order matters: noindex first (crawlable), block later. Never both at once — a blocked page can’t show its noindex.

If you actually want the page indexed

Then the warning is the real problem: your own robots.txt is stopping Google from reading a page you want ranked. Remove the Disallow rule so Google can crawl and index it properly, with a real snippet.

Find every affected URL

Check Search Console: Indexing → Pages → “Indexed, though blocked by robots.txt.” Export the list and decide per URL: keep (unblock) or remove (unblock + noindex). On larger sites these pile up quietly — a site audit like Semrush’s flags robots.txt conflicts and indexing issues across the whole site so you’re not hunting them one URL at a time.

Audit robots.txt & indexing issues

The bottom line

This warning is really a lesson in a distinction most people blur: crawling and indexing are different jobs. robots.txt stops crawling; noindex stops indexing; and a page can’t show its noindex if it’s blocked from crawling. To remove a stuck page: unblock it, add noindex, let Google recrawl, then optionally re-block. The full crawl-vs-index difference — and which tool does what — is in noindex vs robots.txt disallow; for managing crawlers more broadly (including AI bots), see how to block or allow AI crawlers.

If plain, correct SEO answers are your thing, I send them occasionally — subscribe below for those and the free GEO Starter Checklist.

Frequently asked questions

What does "Indexed, though blocked by robots.txt" mean?

It means Google added a URL to its index even though your robots.txt blocked it from crawling the page. Google discovered the URL through links from other pages and indexed it based on those signals, without ever reading the page content — which is why the listing often has no description.

How do I fix "Indexed, though blocked by robots.txt"?

If you want the page removed: first remove the Disallow rule so Google can crawl it, then add a noindex meta tag or X-Robots-Tag header, and let Google recrawl so it can see the noindex and drop the page. Only re-block in robots.txt afterward. If you want the page indexed, simply remove the Disallow rule.

Why does adding robots.txt rules not fix this?

Because robots.txt blocks crawling, not indexing. A blocked page can be indexed from external links, and — critically — Google can never see a noindex tag on a page it is not allowed to crawl. More robots.txt rules keep the page stuck; you must allow crawling for noindex to work.

Does robots.txt remove a page from Google?

No. robots.txt only asks Google not to crawl a page; it does not remove it from the index and can leave URLs listed without a description. To remove a page from Google, allow crawling and use a noindex directive instead.