Two tools. One job people want: “keep this out of Google.” They do different jobs, and using the wrong one is how staging sites and admin folders leak into search as bare URLs.
Use robots.txt to save crawling, not to hide
Disallow infinite filters, internal search results, and cart query strings so Googlebot spends time on pages that can rank. That is crawl-budget hygiene.
Do not Disallow a page you need reliably gone from results. Google can still list a blocked URL if other sites link to it. You get a result with no title and no description. It looks broken because it is: the crawler was never allowed to read a better snippet, or a noindex.
Use noindex when the page must not appear
Allow crawling. Put this in the HTML:
<meta name="robots" content="noindex, follow">
follow still lets equity pass through the links. Use noindex, nofollow only when you also do not want those links counted.
WordPress “Discourage search engines” writes a sitewide noindex. That is the correct tool used on the entire site by accident — see the WordPress audit.
The trap
Blocked in robots.txt and noindex in the template. The crawler never sees the tag. You think you noindexed it. Search Console still shows it as crawled-elsewhere or listed from a link. Remove the Disallow if you need the noindex to work.
Check both in one pass with the noindex checker and the robots.txt tester.
