robots.txt is a plain text file at the root of your domain that tells crawlers where they may go. It's four lines long on most sites, and one wrong character in it can remove you from Google entirely.
Check yours right now
Open yourdomain.com/robots.txt in a browser. You'll see one of three things.
The fatal one
User-agent: *
Disallow: /This blocks every crawler from every URL. It's the default configuration on staging environments, and it reaches production more often than anyone would like to admit. If you see this and your site is meant to be public, that's your problem — delete the Disallow: / line.
The healthy one
User-agent: *
Allow: /
Disallow: /admin
Disallow: /checkout
Sitemap: https://yourdomain.com/sitemap.xmlOpen to crawlers, private areas excluded, sitemap declared. This is what you want.
No file at all
A 404 on robots.txt is not fatal — crawlers assume they're welcome. But you're missing a free opportunity to declare your sitemap, so add one.
The distinction that trips everyone up
robots.txt controls crawling, not indexing. This matters more than it sounds.
If you block a page in robots.txt, Google won't read it — but if other sites link to that URL, Google may still list it in results, showing just the URL with "No information is available for this page". You've achieved the worst of both: it's in the index, and it looks broken.
The rule:
- To keep a page out of search: allow crawling, add
<meta name="robots" content="noindex">. Google must be able to crawl the page to see the tag. - To save crawl budget on infinite filters or faceted URLs: use robots.txt.
Blocking a page in robots.txt and adding noindex is self-defeating — the crawler can never read the noindex.
Other patterns worth catching
- Blocking CSS and JavaScript. An old practice that now actively harms you: Google renders pages, and blocked assets mean it sees a broken layout and may judge your page not mobile-friendly.
- Blocking your own images.
Disallow: /images/removes you from Google Images entirely — for many retailers, a meaningful traffic source. - Googlebot-specific rules. A
User-agent: Googlebotblock overrides the general*group for Google specifically. Read the whole file, not just the first block. - Wrong location. robots.txt only works at the domain root.
/blog/robots.txtis ignored.
After you fix it
Changes aren't instant. Google caches robots.txt for roughly 24 hours. Speed recovery up by submitting the corrected file in Search Console's robots.txt report, then requesting indexing on a few key URLs.
Our robots.txt tester fetches your robots.txt as part of every audit and flags a sitewide block as a critical issue — worth running before you assume the problem lies elsewhere.