Why robots.txt Does Not Remove a Page From Google

The message reads “Indexed, though blocked by robots.txt”, and it looks like a contradiction. It is not. It is the system working exactly as specified, and the confusion comes from assuming one file does two jobs.

Two different controls

robots.txt is a fetch control. It is defined in RFC 9309, it lives at the root of a host, and it tells a compliant crawler which paths it may request. A disallowed URL is never requested.

noindex is an index control. Delivered as a meta robots tag or an X-Robots-Tag header, it tells the engine not to store the page. It is the only directive that reliably removes a page from search results.

They operate at different stages of the pipeline, and neither substitutes for the other.

Why a blocked page still gets listed

A search engine learns that a URL exists from links pointing at it, not from fetching it. If ten pages link to /private/report/ and your robots.txt disallows that path, the engine knows the URL exists, knows what the linking pages’ anchor text says about it, and has never been permitted to look inside.

It can list the URL on that basis. It cannot write a snippet, because it has never seen the content, so the result appears bare — sometimes with a line saying no information is available for this page.

The self-defeating combination

This is the mistake worth remembering:

# robots.txt
User-agent: *
Disallow: /private/

<!-- /private/report/ -->
<meta name="robots" content="noindex">

The crawler is forbidden from fetching /private/report/, so it never reads the noindex. The directive has no effect at all, and the URL stays listed indefinitely.

What to do instead

To remove a page from the index:

1. Allow the fetch. Remove the Disallow rule covering the URL.
2. Serve noindex as a meta tag or X-Robots-Tag header.
3. Wait for a recrawl. You can request one through URL Inspection in Search Console to speed it up.
4. Re-block only after it has dropped, if you also want to save crawl budget.

For anything genuinely confidential, none of this is the right tool. A robots.txt file is public and lists the paths you consider sensitive. Put real access control behind authentication.

For an architectural breakdown of how robots.txt interacts with sitemaps, canonical tags, and noindex, see the controls comparison in Search Engine Basics.