Guides
noindex vs robots.txt Disallow: Why Blocking Doesn't Deindex
On this page
The one-line answer: robots.txt Disallow controls crawling (whether Google reads a page); noindex controls indexing (whether Google lists it). To keep a page out of Google, you need noindex — Disallow alone can leave it indexed. This is the single most common technical-SEO mix-up I see, and getting it wrong is how pages end up stuck in search results you can’t remove. Here’s the difference, clearly.
Crawling vs indexing: two different jobs
These are separate steps, and conflating them causes most of the confusion:
- Crawling = Google reading your page. Controlled by
robots.txt. - Indexing = Google listing your page in search results. Controlled by the
noindexdirective.
A page can be crawled but not indexed, or — the tricky case — indexed but not crawled. They don’t imply each other.
What robots.txt Disallow actually does
Disallow in robots.txt says: “Don’t crawl this.” That’s it. It does not say “don’t index this.”
The consequence trips people up constantly: if other sites (or your own pages) link to a Disallowed URL, Google can still index the URL based on those links — without reading the page. You get a listing, often with no description (“No information is available for this page”). That’s the exact cause of the Indexed, though blocked by robots.txt warning in Search Console.
So: Disallow is a crawl-budget and access tool, not a removal tool.
What noindex actually does
noindex says: “Don’t list this in search results.” You add it as:
- a meta tag in the page’s
<head>:<meta name="robots" content="noindex">, or - an HTTP header:
X-Robots-Tag: noindex(useful for non-HTML files like PDFs).
When Google crawls the page and sees noindex, it removes the page from the index. This is the correct way to keep a page out of Google.
The critical catch: noindex needs crawling
Here’s the trap that ties it all together: for noindex to work, Google must be able to crawl the page — because the directive lives on the page. If you Disallow a URL in robots.txt and put noindex on it, Google can’t read the page, so it never sees the noindex, so it can’t remove the page. The two directives cancel each other out.
Never block a page in robots.txt if you’re relying on its noindex tag.
Which should you use? A simple guide
| Goal | Use |
|---|---|
| Keep a page out of Google’s results | noindex (and let Google crawl it) |
| Stop Google wasting crawl budget on low-value URL patterns | robots.txt Disallow |
| Remove an already-indexed page | noindex first; add Disallow only after it’s dropped |
| Hide a whole section from crawlers you don’t control (e.g. AI bots) | robots.txt Disallow — see blocking AI crawlers |
The removal sequence that actually works: allow crawling → add noindex → wait for Google to drop it → then optionally Disallow to save crawl budget.
The bottom line
robots.txt Disallow and noindex aren’t interchangeable — one governs crawling, the other indexing. If you want a page gone from Google, noindex is the tool, and the page must stay crawlable long enough for Google to see it. Reach for robots.txt when you want to save crawl budget or manage bot access, not to deindex. Master this one distinction and a whole category of Search Console headaches disappears — it’s a cornerstone of technical SEO in the AI era.
If clear technical-SEO answers help, I share them occasionally — subscribe below for those plus the free GEO Starter Checklist.
Frequently asked questions
What is the difference between noindex and robots.txt disallow?
robots.txt Disallow controls crawling — whether Google reads a page. noindex controls indexing — whether Google lists it in search results. Disallow can still leave a URL indexed via external links, so to keep a page out of Google you need noindex, not Disallow.
Does robots.txt disallow remove a page from Google?
No. Disallow only stops crawling; it does not remove a page from the index. A disallowed URL can still be indexed from links pointing to it, often listed without a description. To remove a page, use a noindex directive and allow crawling so Google can see it.
Can I use noindex and robots.txt disallow together?
No — they conflict. noindex lives on the page, so Google must crawl the page to see it. If the page is also disallowed in robots.txt, Google cannot read it and never sees the noindex, so the page stays indexed. Keep the page crawlable until noindex removes it.
How do I remove a page from Google search?
Add a noindex meta tag or X-Robots-Tag header to the page and make sure it is crawlable so Google can see the directive. Once the page drops from the index, you may add a robots.txt Disallow to save crawl budget — but only after noindex has done its job.