Indexing

noindex vs robots.txt: Which One Actually Keeps a Page Out of Google?

Two gates in a dark corridor: one portcullis fully down, one open with a gold ribbon across it
A closed gate stops the crawler. A ribbon the crawler can read is how you ask to be left out of the index.

Two tools. One job people want: “keep this out of Google.” They do different jobs, and using the wrong one is how staging sites and admin folders leak into search as bare URLs.

robots.txt versus noindex robots.txt Disallow noindex (allow crawl) Crawler cannot read the page May still appear as a URL with no snippet Crawler reads the tag Reliable exclusion from search results
If you want a page out of Google, allow crawling and noindex it. Blocking it in robots.txt often produces the worst of both: listed, but empty.

Use robots.txt to save crawling, not to hide

Disallow infinite filters, internal search results, and cart query strings so Googlebot spends time on pages that can rank. That is crawl-budget hygiene.

Do not Disallow a page you need reliably gone from results. Google can still list a blocked URL if other sites link to it. You get a result with no title and no description. It looks broken because it is: the crawler was never allowed to read a better snippet, or a noindex.

Use noindex when the page must not appear

Allow crawling. Put this in the HTML:

<meta name="robots" content="noindex, follow">

follow still lets equity pass through the links. Use noindex, nofollow only when you also do not want those links counted.

WordPress “Discourage search engines” writes a sitewide noindex. That is the correct tool used on the entire site by accident — see the WordPress audit.

The trap

Blocked in robots.txt and noindex in the template. The crawler never sees the tag. You think you noindexed it. Search Console still shows it as crawled-elsewhere or listed from a link. Remove the Disallow if you need the noindex to work.

Check both in one pass with the noindex checker and the robots.txt tester.

Frequently asked

Should I use both robots.txt and noindex together?

Almost never on the same URL. If robots.txt blocks the crawler, it cannot see the noindex tag, so the noindex never applies.

What is the X-Robots-Tag header?

The same directives as a robots meta tag, sent as an HTTP header. Useful for PDFs and non-HTML files that cannot contain a meta tag.

Keep reading